I agree that app testers are good, but ideally unit tests should cover all the codepaths which can be triggered by an app. We can use tools like lcov to verify that the same codepaths are being taken and ensure that identical code is being run in tests as in apps.
Testing in a controlled environment will always be better than random app testing by users, provided that the coverage of the tests is good enough. How many tickets have we had which are not able to be reproduced in an app by a developer after being reported by a user? This is the purpose of good unit testing: to eliminate that gap in coverage by guaranteeing the same code is run every time. With this said, our unit tests are not currently good enough to provide this level of coverage, so adding app testing will be useful for the foreseeable future until we can achieve a significantly better degree of coverage. Targeting the beta release as the point when all compatibility issues are resolved seems reasonable to me; we have never had clear thresholds for what determines each release, so perhaps we can use something like: Alpha - all "urgent" tickets resolved, excluding regressions Beta - all compatibility issues (regressions) resolved After this point we can reassess the release state. If we are going to explicitly include app testing in the release testing process then I think it's important to have a list of apps which are tested along with test cases for those apps. I don't personally use any EFL-based apps in my daily life, so just opening an app and clicking around randomly will probably not be as effective as a more focused plan. On Wed, Jun 27, 2018 at 10:08 PM Hermet Park <hermetp...@gmail.com> wrote: > Additional explanation for the second, > > Unit test, TC, and automation tool is good but not enough. > We need human testers absolutely. They could test apps, detect errors and > then report it. > For the progressive sw, this must be a mandatory, not an optional process. > > For version release(maybe in the alpha step or just before alpha), > We officially announce about app test and list up candidate apps for the > test. > > Basically each app maintainers(actual app developers) could take cover > their apps. > In case of absence of maintainer(but still the app is good to maintain), we > can delegate a tester for the app. > Even if 1 app, 1 tester is good. People who are familiar with the app is > better. > > For the delegated case, testers can try all possible functionalities of the > app. If they find any wrong, weird behaviors, > they can compare the app working on the previous efl version. > > If a wrong behavior is only showed on new efl version, they can list up the > issue and share with community officially. > > If efl developers approve the issue is about compatibilty issue, they must > fix it before beta. > > > On Wed, Jun 27, 2018 at 9:18 PM, Mike Blumenkrantz < > michael.blumenkra...@gmail.com> wrote: > > > It seems like you have two main points here: > > > > 1) The review process for components should require the approval of > > "maintainers", or people who are experts in a given area, and not just > > anyone with commit access. > > > > I agree with this idea. While it would certainly be nice if everyone > could > > review every patch, not everyone has the same level of expertise in every > > part of the codebase, and having someone without enough knowledge of the > > code doing review on it can lead to problems. > > I think the next step for this would be to create a proposal with some > > initial "maintainers" for given parts of the codebase and set up herald > > rules to automatically add them as reviewers when patches are submitted > for > > their areas. Then we can see which areas need more coverage? > > > > > > 2) Testing is still inadequate, and applications need to work with > versions > > of EFL that are about to ship. > > > > This is also a good point. There are cases where applications are broken > by > > changes in EFL, and we should avoid this. The problem, however, is that > > application-based testing will always be a bit random due to differences > in > > user configs and distro setups. > > We should resolve this by continuing to improve our unit testing as well > as > > working to integrate Exactness into the test process. As coverage of > these > > two types of testing increase, the difference between "testing by running > > an app" and "testing by running 'make check'" will lessen until > eventually > > only 'make check' should be needed. This also requires some work from app > > developers: any time their app is broken, they should ensure there is a > > clear report, and a unit test of some kind should be added to ensure that > > the issue does not occur again. > > > > On Mon, Jun 25, 2018 at 11:54 PM Hermet Park <hermetp...@gmail.com> > wrote: > > > > > Thankfully, Xavi Artigas fixed. Even though unstable but it's > compilable > > on > > > devel branch now. > > > > > > Actually, efl has been bad at compatibility so far. > > > After 1 year leaving, Enventor has totally collapsed which was stable > > > enough. > > > It's not about interface nor compilation but behaviors. > > > UIs is too fragile now, entry and focus in Enventor showed me the worst > > > compatibility among that. > > > It's too shame. > > > > > > Since efl is a very huge system. many committers are not easy to > > understand > > > whole bunch of code and history. > > > > > > I agree with zmike, bu5hm4n and other people about review process. We > > need > > > more strict patch review. > > > But I'd rather say, it doesn't mean just review but the real review by > > > actual maintainers. > > > Submitting patches without actual maintainer's review shoudln't be > > > acceptable even if they are committer. > > > If a non-maintainer reviews code, verifies it, situation won't be > > > different. > > > > > > In my experience, unstable efl has not come from the misunderstanding > of > > > programming language, eye-catching compatibility breakage, crash and > > simple > > > logic change inside the function. > > > Sometimes we do mistake but it's not too critical. We can easily find > > them > > > and fix them soon by compiler, code analizyer, test apps and > developers. > > > > > > What's on the stake? > > > > > > Unstable efl has coming from the misunderstanding of code history, > whole > > > logic sequence in real complex usage and api concept changes. > > > > > > Obviously, we need official designated maintainers in efl, at least > those > > > maintaining parts should not be broken any more. > > > > > > Plus, as we know, UI is not a single functional stuff. It's applicable. > > > unit test or single function test is not enough. > > > Even we human cannot imagine or expect how our changes will bring > broken > > > behaviors from this complex world. > > > > > > few but still, we have those efl apps and active app maintainers- rage, > > > terminology, eruler, ephoto and enventor etc > > > Actually they are not enough number of unit but unless we try to keep > > them > > > alive, well, who wanna untrustworthy library and write apps using it? > > > > > > I wish efl developers should not satisfy with just unit test / test > apps > > > such as elementary_test. > > > Before a new version release, additionally, we should get approval by > > more > > > practical apps whether they have any compatibility issues or not. > > > > > > > > > On Tue, Jun 26, 2018 at 11:11 AM, Simon Lees <sfl...@suse.de> wrote: > > > > > > > > > > > > > > > On 26/06/18 04:36, Mike Blumenkrantz wrote: > > > > > Hi, > > > > > > > > > > Do not attempt to use Eflete. > > > > > > > > > > I was attempting to examine a related issue from phab today on my > > test > > > > > machine and Eflete somehow managed to delete nearly EVERYTHING from > > my > > > > home > > > > > directory. All my .directories, all my source trees, everything > > > deleted. > > > > > > > > > > Fortunately, this was only a testing machine and nothing other than > > > some > > > > > local configs were lost. > > > > > > > > > > DO NOT USE EFLETE. > > > > > > > > > > > > > > > Regards, > > > > > Mike > > > > > > > > Last time I tried to use it, it wouldn't build against the current > > > > stable efl release so I gave up, which is a shame because when it was > > > > working well it was a really useful tool. > > > > > > > > -- > > > > > > > > Simon Lees (Simotek) http://simotek.net > > > > > > > > Emergency Update Team keybase.io/simotek > > > > SUSE Linux Adelaide Australia, UTC+10:30 > > > > GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B > > > > > > > > > > > > ------------------------------------------------------------ > > > > ------------------ > > > > Check out the vibrant tech community on one of the world's most > > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > _______________________________________________ > > > > enlightenment-devel mailing list > > > > enlightenment-devel@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > > > > > > > > > > > > > > > > -- > > > Regards, Hermet > > > > > > ------------------------------------------------------------ > > ------------------ > > > Check out the vibrant tech community on one of the world's most > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > _______________________________________________ > > > enlightenment-devel mailing list > > > enlightenment-devel@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > > > > ------------------------------------------------------------ > > ------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > _______________________________________________ > > enlightenment-devel mailing list > > enlightenment-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > > > > > > -- > Regards, Hermet > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > enlightenment-devel mailing list > enlightenment-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
