https://github.com/matomo-org/matomo/commit/5d2dab75e3654126ed95ea72b58b4c225ac59492 - Mouve

Site d'origine

Permalink
Browse files
Merge branch 3.x-dev into 4.x-dev (#15543)
* Updates search engine and social definitions (#15384)

* updates device detector to latest release (#15388)

* updates device detector to latest release

* updates tests

* translation update (#15389)

* Fix Could not get the lock for ID, when creating a site (#15401)

* Lock key start

* do not empty key lock

Co-authored-by: Thomas Steur <tsteur@users.noreply.github.com>

* 3.13.1

* submodule updates

* Use correct name in update available message (#15423)

* Fix removing user capabilities (#15422)

* Order of implode() args, avoid E_NOTICE in PHP7.4 (#15428)

* Fixes possible php warning in visitor log (#15442)

* silence is_executable call (#15446)

* Make sure geolocation admin experience is consistent if user is not using GeoIp2 plugin. (#15447)

* Fix referrers test. (#15448)

* Ensure to close visitor popover correctly (#15443)

* Fixes possible warning (#15453)

* Forward instance_id from local config when reseting config during tests. (#15445)

* Add event that allows plugins to disable archiving for certain periods/sites if they want. (#15457)

* Add event that allows plugins to disable archiving for certain periods/sites if they want.

* apply review feedback

* Fix possible warning for columns without index (#15467)

* Day range archiving issue (#15462)

* Improve lock ID check for max length (#15407)

Better patch for #15401 which was merged last minute...

This way it always works even when someone calls `acquireLock` directly instead of `execute`

Pushing this for now into 3.x-dev but can also put it into 4.x-dev directly but then there might be merge conflicts when merging 3.x-dev into 4.x-dev

* Use SameSite none for session token when embedded into iframe (#15439)

* Make sure tracking works in IE9 and lower (#15480)

* Mention Joomla install FAQ (#15481)

* Make sparklines work when mbstring extension is not installed (#15489)

1) Too few arguments to function mb_strtolower(), 1 passed in matomo/vendor/davaxi/sparkline/src/Sparkline/StyleTrait.php on line 129 and exactly 2 expected

2) mb_strlen is not defined

* update screenshots (#15488)

* 3.13.2-rc1

* Use safemode when running CLI commands (#15472)

* update icons submodule (#15490)

* update icons submodule

* update UI tests

* Fix possible undefined index notice (#15502)

* Use latest davaxi/sparkline release (#15464)

* translation update

* submodule updates

* Fix deprecation notice (#15530)

see #15467 (comment)

* 3.13.2-rc2

* update cache component (#15536)

*  fixes copy dashboard to user for more than 100 users (#15538)

cherry picking  #15424  to fix #15420 in 3.x-dev

* Add missing return statement. (#15539)

* 3.13.2

* update tests

* update tests

Co-authored-by: Matthieu Aubry <mattab@users.noreply.github.com>
Co-authored-by: Thomas Steur <tsteur@users.noreply.github.com>
Co-authored-by: Peter Upfold <pgithub@upfold.org.uk>
Co-authored-by: diosmosis <diosmosis@users.noreply.github.com>
Co-authored-by: Lukas Winkler <github@lw1.at>
  • Loading branch information
6 people committed Feb 11, 2020
1 parent ab1e700 commit 5d2dab75e3654126ed95ea72b58b4c225ac59492
Showing with 480 additions and 133 deletions.
  1. +2 −2 composer.json
  2. +11 −11 composer.lock
  3. +5 −0 console
  4. +3 −2 core/Archive.php
  5. +4 −2 core/ArchiveProcessor/PluginsArchiver.php
  6. +16 −4 core/ArchiveProcessor/Rules.php
  7. +27 −2 core/Archiver/Request.php
  8. +1 −1 core/CliMulti/CliPhp.php
  9. +7 −7 core/Concurrency/Lock.php
  10. +15 −0 core/Console.php
  11. +19 −2 core/CronArchive.php
  12. +1 −1 core/DataAccess/ArchiveSelector.php
  13. +1 −1 core/FrontController.php
  14. +4 −4 core/Plugin/Report.php
  15. +11 −0 core/Session.php
  16. +1 −1 core/Session/SessionAuth.php
  17. +1 −1 core/Tracker/Model.php
  18. +1 −1 core/Tracker/Settings.php
  19. +2 −2 js/piwik.js
  20. +4 −4 js/piwik.min.js
  21. +2 −2 libs/Zend/Session.php
  22. +16 −3 libs/upgradephp/upgrade.php
  23. +4 −4 matomo.js
  24. +4 −4 piwik.js
  25. +7 −0 ...I/tests/System/expected/test_AutoSuggestAPITest_browserName__API.getSuggestedValuesForSegment.xml
  26. +1 −0 ...System/expected/test_AutoSuggestAPITest_operatingSystemName__API.getSuggestedValuesForSegment.xml
  27. +17 −3 plugins/CoreAdminHome/javascripts/optOut.js
  28. +1 −1 plugins/CoreHome/templates/_headerMessage.twig
  29. +3 −2 plugins/CoreVisualizations/Visualizations/HtmlTable.php
  30. +2 −2 plugins/Dashboard/tests/UI/expected-screenshots/Dashboard_loaded_token_auth.png
  31. +2 −2 plugins/Dashboard/tests/UI/expected-screenshots/Dashboard_removed.png
  32. +5 −2 plugins/Live/Visualizations/VisitorLog.php
  33. +21 −8 plugins/Live/javascripts/visitorProfile.js
  34. +1 −1 plugins/Morpheus/icons
  35. +2 −2 ...PrivacyManager/tests/UI/expected-screenshots/PrivacyManager_gdpr_tools_delete_visit_cancelled.png
  36. +2 −2 ...ivacyManager/tests/UI/expected-screenshots/PrivacyManager_gdpr_tools_delete_visit_unconfirmed.png
  37. +2 −2 ...s/PrivacyManager/tests/UI/expected-screenshots/PrivacyManager_gdpr_tools_enrich_segment_by_ip.png
  38. +2 −2 plugins/PrivacyManager/tests/UI/expected-screenshots/PrivacyManager_gdpr_tools_uncheck_one_visit.png
  39. +7 −7 plugins/Referrers/tests/System/ApiTest.php
  40. +1 −0 plugins/SitesManager/templates/_trackingCodeEmail.twig
  41. +1 −0 plugins/SitesManager/templates/siteWithoutData.twig
  42. +6 −0 plugins/UserCountry/Controller.php
  43. +2 −1 plugins/UserCountry/lang/el.json
  44. +2 −1 plugins/UserCountry/lang/en.json
  45. +2 −1 plugins/UserCountry/lang/es-ar.json
  46. +2 −1 plugins/UserCountry/lang/pt-br.json
  47. +2 −1 plugins/UserCountry/lang/sq.json
  48. +2 −1 plugins/UserCountry/lang/tr.json
  49. +11 −2 plugins/UserCountry/templates/adminIndex.twig
  50. +1 −1 plugins/UsersManager/angularjs/capabilities-edit/capabilities-edit.component.html
  51. +15 −0 tests/PHPUnit/Framework/Fixture.php
  52. +10 −1 tests/PHPUnit/Framework/Mock/TestConfig.php
  53. +132 −0 tests/PHPUnit/System/ConsoleTest.php
  54. +1 −1 tests/PHPUnit/System/expected/test_apiGetReportMetadata__API.getSegmentsMetadata.xml
  55. +28 −0 tests/PHPUnit/Unit/Archiver/RequestTest.php
  56. +2 −2 tests/UI/expected-screenshots/Comparison_segmented_visitorlog.png
  57. +2 −2 tests/UI/expected-screenshots/EmptySite_emptySiteDashboard.png
  58. +2 −2 tests/UI/expected-screenshots/OneClickUpdate_login.png
  59. +2 −2 tests/UI/expected-screenshots/Theme_home.png
  60. +2 −2 tests/UI/expected-screenshots/UIIntegrationTest_dashboard1.png
  61. +2 −2 tests/UI/expected-screenshots/UIIntegrationTest_ecommerce_log.png
  62. +2 −2 tests/UI/expected-screenshots/UIIntegrationTest_ecommerce_log_segmented.png
  63. +2 −2 tests/UI/expected-screenshots/UIIntegrationTest_segmented_visitorlog.png
  64. +2 −2 tests/UI/expected-screenshots/UIIntegrationTest_visitor_profile_popup.png
  65. +2 −2 tests/UI/expected-screenshots/UIIntegrationTest_visitors_realtime_map.png
  66. +2 −2 tests/UI/expected-screenshots/UIIntegrationTest_visitors_software.png
  67. +2 −2 tests/UI/expected-screenshots/UIIntegrationTest_widgetize_ecommercelog.png
  68. +1 −1 tests/travis
@@ -37,8 +37,8 @@
"matomo/ini": "~2.0",
"matomo/matomo-php-tracker": "~2.0",
"matomo/network": "~2.0",
"matomo/referrer-spam-blacklist": "~1.0",
"matomo/searchengine-and-social-list": "~1.0",
"matomo/referrer-spam-blacklist": "~3.0",
"matomo/searchengine-and-social-list": "~3.0",
"monolog/monolog": "~1.11",
"mustangostang/spyc": "~0.6.0",
"pear/pear_exception": "~1.0.0",

Some generated files are not rendered by default. Learn more.

@@ -1,5 +1,8 @@
#!/usr/bin/env php
<?php

use Piwik\FrontController;

if (!defined('PIWIK_DOCUMENT_ROOT')) {
define('PIWIK_DOCUMENT_ROOT', dirname(__FILE__) == '/' ? '' : dirname(__FILE__));
}
@@ -23,5 +26,7 @@ if (!defined('PIWIK_ENABLE_ERROR_HANDLER') || PIWIK_ENABLE_ERROR_HANDLER) {
Piwik\ExceptionHandler::setUp();
}

FrontController::setUpSafeMode();

$console = new Piwik\Console();
$console->run();
@@ -649,10 +649,11 @@ private function cacheArchiveIdsAfterLaunching($archiveGroups, $plugins)
foreach ($this->params->getIdSites() as $idSite) {
$site = new Site($idSite);

if ($period->getLabel() === 'day'
if ($period->getLabel() === 'day'
&& !$this->params->getSegment()->isEmpty()
&& Common::getRequestVar('skipArchiveSegmentToday', 0, 'int')
&& $period->getDateStart()->toString() == Date::factory('now', $site->getTimezone())->toString()) {
&& $period->getDateStart()->toString() == Date::factory('now', $site->getTimezone())->toString()
) {

Log::debug("Skipping archive %s for %s as segment today is disabled", $period->getLabel(), $period->getPrettyString());
continue;
@@ -160,13 +160,15 @@ public function callAggregateAllPlugins($visits, $visitsConverted, $forceArchivi
try {
self::$currentPluginBeingArchived = $pluginName;

$period = $this->params->getPeriod()->getLabel();

$timer = new Timer();
if ($this->shouldAggregateFromRawData) {
Log::debug("PluginsArchiver::%s: Archiving day reports for plugin '%s'.", __FUNCTION__, $pluginName);
Log::debug("PluginsArchiver::%s: Archiving $period reports for plugin '%s' from raw data.", __FUNCTION__, $pluginName);

$archiver->callAggregateDayReport();
} else {
Log::debug("PluginsArchiver::%s: Archiving period reports for plugin '%s'.", __FUNCTION__, $pluginName);
Log::debug("PluginsArchiver::%s: Archiving $period reports for plugin '%s' using reports for smaller periods.", __FUNCTION__, $pluginName);

$archiver->callAggregateMultipleReports();
}
@@ -226,9 +226,21 @@ public static function isArchivingDisabledFor(array $idSites, Segment $segment,
return !$isArchivingEnabled;
}

public static function isRequestAuthorizedToArchive()
public static function isRequestAuthorizedToArchive(Parameters $params = null)
{
return Rules::isBrowserTriggerEnabled() || SettingsServer::isArchivePhpTriggered();
$isRequestAuthorizedToArchive = Rules::isBrowserTriggerEnabled() || SettingsServer::isArchivePhpTriggered();

if (!empty($params)) {
/**
* @ignore
*
* @params bool &$isRequestAuthorizedToArchive
* @params Parameters $params
*/
Piwik::postEvent('Archiving.isRequestAuthorizedToArchive', [&$isRequestAuthorizedToArchive, $params]);
}

return $isRequestAuthorizedToArchive;
}

public static function isBrowserTriggerEnabled()
@@ -293,11 +305,11 @@ public static function isSegmentPreProcessed(array $idSites, Segment $segment)
*
* @return string[]
*/
public static function getSelectableDoneFlagValues($includeInvalidated = true)
public static function getSelectableDoneFlagValues($includeInvalidated = true, Parameters $params = null)
{
$possibleValues = array(ArchiveWriter::DONE_OK, ArchiveWriter::DONE_OK_TEMPORARY);

if (!Rules::isRequestAuthorizedToArchive()
if (!Rules::isRequestAuthorizedToArchive($params)
&& $includeInvalidated
) {
//If request is not authorized to archive then fetch also invalidated archives
@@ -31,7 +31,7 @@ class Request
*/
public function __construct($url)
{
$this->url = $url;
$this->setUrl($url);
}

public function before($callable)
@@ -68,9 +68,34 @@ public function setUrl($url)
}

public function changeDate($newDate)
{
$this->changeParam('date', $newDate);
}

public function makeSureDateIsNotSingleDayRange()
{
// TODO: revisit in matomo 4
// period=range&date=last1/period=range&date=previous1 can cause problems during archiving due to Parameters::isDayArchive()
if (preg_match('/[&?]period=range/', $this->url)) {
if (preg_match('/[&?]date=last1/', $this->url)) {
$this->changeParam('period', 'day');
$this->changeParam('date', 'today');
} else if (preg_match('/[&?]date=previous1/', $this->url)) {
$this->changeParam('period', 'day');
$this->changeParam('date', 'yesterday');
} else if (preg_match('/[&?]date=([^,]+),([^,&]+)/', $this->url, $matches)
&& $matches[1] == $matches[2]
) {
$this->changeParam('period', 'day');
$this->changeParam('date', $matches[1]);
}
}
}

public function changeParam($name, $newValue)
{
$url = $this->getUrl();
$url = preg_replace('/([&?])date=[^&]*/', '$1date=' . $newDate, $url);
$url = preg_replace('/([&?])' . preg_quote($name) . '=[^&]*/', '$1' . $name . '=' . $newValue, $url);
$this->setUrl($url);
}
}
@@ -86,7 +86,7 @@ private function isValidPhpType($path)

private function getPhpCommandIfValid($path)
{
if (!empty($path) && is_executable($path)) {
if (!empty($path) && @is_executable($path)) {
if (0 === strpos($path, PHP_BINDIR) && $this->isValidPhpType($path)) {
return $path;
}
@@ -50,13 +50,6 @@ public function getAllAcquiredLockKeys()

public function execute($id, $callback)
{
if (Common::mb_strlen($id) > self::MAX_KEY_LEN) {
// Lock key might be too long for DB column, so we hash it but leave the start of the original as well
// to make it more readable
$md5Len = 32;
$id = Common::mb_substr($id, 0, self::MAX_KEY_LEN - $md5Len - 1) . md5($id);
}

$i = 0;
while (!$this->acquireLock($id)) {
$i++;
@@ -76,6 +69,13 @@ public function acquireLock($id, $ttlInSeconds = 60)
{
$this->lockKey = $this->lockKeyStart . $id;

if (Common::mb_strlen($this->lockKey) > self::MAX_KEY_LEN) {
// Lock key might be too long for DB column, so we hash it but leave the start of the original as well
// to make it more readable
$md5Len = 32;
$this->lockKey = Common::mb_substr($id, 0, self::MAX_KEY_LEN - $md5Len - 1) . md5($id);
}

$lockValue = substr(Common::generateUniqId(), 0, 12);
$locked = $this->backend->setIfNotExists($this->lockKey, $lockValue, $ttlInSeconds);

@@ -77,6 +77,21 @@ public function renderException($e, $output)
}

public function doRun(InputInterface $input, OutputInterface $output)
{
try {
return $this->doRunImpl($input, $output);
} catch (\Exception $ex) {
try {
FrontController::generateSafeModeOutputFromException($ex);
} catch (\Exception $ex) {
// ignore, we re-throw the original exception, not a wrapped one
}

throw $ex;
}
}

private function doRunImpl(InputInterface $input, OutputInterface $output)
{
if ($input->hasParameterOption('--xhprof')) {
Profiler::setupProfilerXHProf(true, true);
@@ -1104,7 +1104,16 @@ private function archiveReportsFor($idSite, $period, $date, $archiveSegments, Ti
return Request::ABORT;
}

$urlBefore = $request->getUrl();
$request->changeDate($newDate);
$request->makeSureDateIsNotSingleDayRange();

// check again if we are already archiving the URL since we just changed it
if ($request->getUrl() !== $urlBefore
&& $self->isAlreadyArchivingSegment($request->getUrl(), $idSite, $period, $segment)
) {
return Request::ABORT;
}

$this->logArchiveWebsite($idSite, $period, $newDate);
});
@@ -1972,9 +1981,17 @@ private function getUrlsWithSegment($idSite, $period, $date)
return Request::ABORT;
}

$url = $request->getUrl();
$url = preg_replace('/([&?])date=[^&]*/', '$1date=' . $newDate, $url);
$urlBefore = $request->getUrl();
$url = preg_replace('/([&?])date=[^&]*/', '$1date=' . $newDate, $urlBefore);
$request->setUrl($url);
$request->makeSureDateIsNotSingleDayRange();

// check again if we are already archiving the URL since we just changed it
if ($request->getUrl() !== $urlBefore
&& $self->isAlreadyArchivingSegment($request->getUrl(), $idSite, $period, $segment)
) {
return Request::ABORT;
}

$processedSegmentCount++;
$logger->info(sprintf(
@@ -71,7 +71,7 @@ public static function getArchiveIdAndVisits(ArchiveProcessor\Parameters $params
$plugins = array("VisitsSummary", $requestedPlugin);

$doneFlags = Rules::getDoneFlags($plugins, $segment);
$doneFlagValues = Rules::getSelectableDoneFlagValues($includeInvalidated);
$doneFlagValues = Rules::getSelectableDoneFlagValues($includeInvalidated, $params);

$results = self::getModel()->getArchiveIdAndVisits($numericTable, $idSite, $period, $dateStartIso, $dateEndIso, $minDatetimeIsoArchiveProcessedUTC, $doneFlags, $doneFlagValues);

@@ -106,7 +106,7 @@ private static function generateSafeModeOutputFromError($lastError)
* @param Exception $e
* @return string
*/
private static function generateSafeModeOutputFromException($e)
public static function generateSafeModeOutputFromException($e)
{
StaticContainer::get(LoggerInterface::class)->error('Uncaught exception: {exception}', [
'exception' => $e,
@@ -496,8 +496,8 @@ protected function getMetricsDocumentation()
} elseif ($metric instanceof Metric) {
$name = $metric->getName();
$metricDocs = $metric->getDocumentation();
if (empty($metricDocs)) {
$metricDocs = @$translations[$name];
if (empty($metricDocs) && !empty($translations[$name])) {
$metricDocs = $translations[$name];
}

if (!empty($metricDocs)) {
@@ -513,8 +513,8 @@ protected function getMetricsDocumentation()
} elseif ($processedMetric instanceof Metric) {
$name = $processedMetric->getName();
$metricDocs = $processedMetric->getDocumentation();
if (empty($metricDocs)) {
$metricDocs = @$translations[$name];
if (empty($metricDocs) && !empty($translations[$name])) {
$metricDocs = $translations[$name];
}

if (!empty($metricDocs)) {
@@ -171,6 +171,17 @@ public static function isSessionStarted()
return self::$sessionStarted;
}

public static function getSameSiteCookieValue()
{
$config = Config::getInstance();
$general = $config->General;
if (!empty($general['enable_framed_pages']) && ProxyHttp::isHttps()) {
return 'None';
}

return 'Lax';
}

/**
* Write cookie header. Similar to the native setcookie() function but also supports
* the SameSite cookie property.
@@ -195,7 +195,7 @@ private function updateSessionExpireTime(SessionFingerprint $sessionFingerprint)
$sessionParams['domain'],
$sessionParams['secure'],
$sessionParams['httponly'],
'lax'
Session::getSameSiteCookieValue()
);

// ...and we also update the expiration time stored server side so we can prevent expired sessions from being reused
@@ -346,7 +346,7 @@ public function updateAction($idLinkVa, $valuesToUpdate)

list($updateParts, $sqlBind) = $this->fieldsToQuery($valuesToUpdate);

$parts = implode($updateParts, ', ');
$parts = implode(', ', $updateParts);
$table = Common::prefixTable('log_link_visit_action');

$sqlQuery = "UPDATE $table SET $parts WHERE idlink_va = ?";
@@ -40,7 +40,7 @@ public function getConfigId(Request $request, $ipAddress)
$deviceDetector = StaticContainer::get(DeviceDetectorFactory::class)->makeInstance($userAgent);
$aBrowserInfo = $deviceDetector->getClient();

if ($aBrowserInfo['type'] != 'browser') {
if (empty($aBrowserInfo['type']) || 'browser' !== $aBrowserInfo['type']) {
// for now only track browsers
unset($aBrowserInfo);
}

0 comments on commit 5d2dab7

Please sign in to comment.

level de log mis à 1

Récupération du document https://github.com/matomo-org/matomo/commit/5d2dab75e3654126ed95ea72b58b4c225ac59492.

Récupération de l'URL https://github.com/matomo-org/matomo/commit/5d2dab75e3654126ed95ea72b58b4c225ac59492...

Raccourcis

Commandes

Fermer