Add code testing via phpunit and/or cypress

[parapente]

This is an effort to add automated testing of the code to ensure that the extension works whenever new patches are added to the codebase or the joomla framework changes.

There are few phpunit tests (ported from older tests that were present in the codebase) but almost all fail for now. It will need quite some effort to refactor the extension code to be able to test it effectivly using phpunit.

Cypress on the other hand can help us test the extension as it is running on an joomla instance like we would manually. I started adding some basic tests that are run in batches. I have created folders for each part of the package in /cypress/e2e that eventually will be populated with the tests.

I will add a checklist for everything that needs to be done and of course the last step would be to add github actions to run the tests whenever we push code to master.

[JLTRY]

Hello it seems to be a big work.
I wanted to try to use the tests.
Are there ready yet ?
I see that the file FOR_DEVELOPERS.md was modified for PHPUnit
Is it usable ?
I see also docker files for joomla cypress novnc , what are the files used for ?

[parapente]

For phpunit:

• You need to run composer install (or composer update if you get a warning about the lock file)
• composer test (to run the tests)

For cypress:

• Requirement: You need to have docker compose available on your system (with docker desktop on windows and mac, with docker desktop or docker repository in linux) Install instructions
• Copy the .env.4.4.example to .env (This will add joomla 4.4 with all the necessary requirements for the tests. The other 2 example files are for joomla 5.4 and 6 respectively. I am still testing with 4.4)
• docker compose --profile testing up -d
• If all went well you now can open http://localhost:8090 and connect to novnc which will have cypress desktop opened. You select end 2 end tests and the browser of your choice (I have only tested chrome for now) and then select a spec file to run the tests. It only contains a small amount of tests but I will eventually add a lot more.
• Using http://localhost:8080 you can access the joomla site and if you run docker compose up -d phpmyadmin you will have on http://localhost:8888 phpmyadmin available to access the database.

[JLTRY]

@parapente for phpunit I have an error at st step 1)

Tests\Unit\Helper\AttachmentsInstallTest::testInstallAttachmentsPermissionsMethodExists
PHPUnit\Framework\MockObject\Generator\ReflectionException: Class "Joomla\CMS\User\User" does not exist

[JLTRY]

I fixed last error by adding link from temp\joomla\libraries\src to joomla 4.x library src path

[JLTRY]

Next error is

Tests\Unit\Helper\AttachmentsPermissionsTest::testGetActionsSuperAdmin
Error: Call to a member function load() on false

F:\Sites\site OVH JLT local\joomla_4.0\libraries\src\User\User.php:819`

[parapente]

@JLTRY yes that is the error I'm getting too. The summary gives:

Tests: 222, Assertions: 265, Errors: 75, Deprecations: 16

but most of those that pass as quite trivial tests. Another thing that will be of importance if we continue to work with phpunit is that the joomla framework packages have different numbering from the joomla cms releases:

Joomla Framework	Joomla CMS
2.x	4.x
3.x	5.x
4.x	6.x

So for the packages that I have added as requirements in composer.json we should use the joomla 6 code in /temp/joomla.

[JLTRY]

I have made a link to joomla_6 libraries/src. It still fails
I commented the lines in bootstrap

/*if (class_exists('\\Joomla\\CMS\\Factory')) {
   // If running within a Joomla installation, use the Joomla factory
   // This is for when tests are run from within a Joomla instance
} else {
   // Setup Joomla framework for standalone testing
   // This is where we'd initialize mock services if needed
}*/

I have got this error
Error: mysqli object is not fully initialized
Let me know if you go further

[JLTRY]

@JLTRY If you get and empty desktop in novnc there is probably an error during cypress desktop startup. You can see the logs if you run docker compose logs cypress. You can retry to run cypress via docker compose restart cypress or docker compose up -d cypress.

Do you run cypress under windows /docker or under linux / docker ?

[parapente]

I am using linux / docker but using docker should give you the same result both on linux and windows. On windows is a little more complicated to get started with docker and a little less performant but it should be the same.

There is another way to run the cypress tests but it will require you to do some manual configuration. Instead of running docker compose --profile testing up -d you can omit the profile option and just give docker compose up -d. This will start only the apache/mariadb servers and you will have the website at http://localhost:8080. Then you can install the cypress desktop app in windows and configure it to find the application and the joomla url. The configuration file for the project is the file cypress.config.js in the root of the project. Please don't push changes to this file on the repository.

Cypress has two modes of running: headless (which just runs the tests and doesn't open an application window) and desktop. For now I am using the desktop mode to write the tests and when they are ready, using the same config we could add a github action to run the tests automatically on every push we make to the repository.

[JLTRY]

@parapente I was able to launch at least one test JoomlasReady
I had to make 1 change : install locally the packages with npm install to have them in cypress container
I have an other issue database timeout
Your right on windows it is very slow with all this dockers
I will try with cypress desktop

[parapente]

docker desktop on windows uses wsl which has the tendency of halfing your pc's memory even if wsl does nothing (half for windows, half for wsl). If your pc has only 8gb of ram things will be extra tight. See if the solution found here helps: https://www.reddit.com/r/docker/comments/135p45t/wsl_using_way_too_much_ram/
apache + mariadb + cypress for tests will need at least 4gb of ram (could run with less but probably will run slow).

I will change the tests slightly as the tests right now empty the database at each run which is not recommended. That will probably fix the issue you have right now as it seems to fail to empty the database.

[parapente]

@JLTRY I managed to make the tests run correctly:

PHPUnit 10.5.58 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.4.11
Configuration: /home/DIDE/teo/Projects/attachments/phpunit.xml

..........WW.WWWWWWWWW.........................................  63 / 222 ( 28%)
............................................................... 126 / 222 ( 56%)
............................................................... 189 / 222 ( 85%)
.................................                               222 / 222 (100%)

Time: 00:00.125, Memory: 14.00 MB

OK, but there were issues!
Tests: 222, Assertions: 440, Warnings: 1, Deprecations: 1.

I am not 100% happy with the state of the code but it at least works. I will review the code on the weekend to untangle the mess I created just to make it work. Joomla has many interconnected parts that makes it difficult to run such tests without running a web server & database. Feel free to test on your end for any issues.

[JLTRY]

@parapente
I have the integration tests that are failing

1) Tests\Integration\Component\Admin\Import\ImportAttachmentsTest
Joomla\Database\Exception\UnsupportedAdapterException: PDO Extension is not available.

F:\Sites\repos\attachments\vendor\joomla\database\src\Pdo\PdoDriver.php:184
F:\Sites\repos\attachments\vendor\joomla\database\src\Sqlite\SqliteDriver.php:80
F:\Sites\repos\attachments\vendor\joomla\database\src\Sqlite\SqliteDriver.php:361
F:\Sites\repos\attachments\vendor\joomla\test\src\DatabaseTestCase.php:55

Any advice ?

[parapente]

@JLTRY You need to install the php sqlite extension to be able to run the tests. By the way I forgot to mention that there are some data missing from the repository that were used to initialize some tables with data. They are not really necessary but this is the reason you will see some hard coded values in the code. I will need to extract those values in a seperate structure.

[JLTRY]

Runtime: PHP 8.3.3
Configuration: F:\Sites\repos\attachments\phpunit.xml

..........WW.WWWWWWWWW......................................... 63 / 222 ( 28%)
............................................................... 126 / 222 ( 56%)
............................................................... 189 / 222 ( 85%)
................................. 222 / 222 (100%)

Time: 00:03.729, Memory: 16.00 MB

OK, but there were issues!
Tests: 222, Assertions: 440, Warnings: 1, Deprecations: 1.
same as you !!!

[JLTRY]

@parapente To be noticed that we can tell phpunit to be more verbose by setting
displayDetailsOnTestsThatTriggerDeprecations="true"
displayDetailsOnTestsThatTriggerErrors="true"
displayDetailsOnTestsThatTriggerNotices="true"
displayDetailsOnTestsThatTriggerWarnings="true"
displayDetailsOnPhpunitDeprecations="true"
For Warning I have got always the same one
`

1. F:\Sites\site OVH JLT local\joomla_5.0\libraries\src\Plugin\PluginHelper.php:190
   foreach() argument must be of type array|object, null given

Triggered by:

• Tests\Integration\Component\Admin\Import\ImportAttachmentsTest::testImportAttachmentsFromCSVFile#0
  F:\Sites\repos\attachments\tests\Integration\Component\Admin\Import\ImportAttachmentsTest.php:154
  `
  It is releated to no (null) plugins found in database

[parapente]

I did check the warnings/deprecations using the switches --display-warnings and --display-deprecations but the ones that are left don't really matter much. They are not in our code but due to the way the code is running some joomla function fails to return an array and that is why we get the warning/deprecation. While refactoring the test code I will revisit the warning in case I can make the test skip this joomla plugin function.

[parapente]

@JLTRY I tried to rebuild the joomla_db.xml that was missing from the codebase and use sqlite for querying 'real' data instead of mocking everything. This adds more tests to the phpunit suite. I will continue with cypress tests next.

[JLTRY]

I checked the unit tests and it is OK for me