Denis, I can see the download.cgi script in SVN. Am I missing something? On Mon, Oct 2, 2017 at 5:31 PM, Denis Magda <dma...@apache.org> wrote:
> Dmitriy, > > That’s the rule. See replies in the ticket [1] > > *Background: the TLP server is already pretty darned busy just serving > *static* sites. Dynamic operation for near-200 PMCs would bury the machine. > Our policy of "static-only websites" has been in place since the Foundation > started* > > Download scripts seem to be the only exception and we as PMC don’t even > have access to them. > > If you want to keep pushing this direction let’s craft a message to Greg > and Daniel directly. I don’t know what else to ask for here rather than a > virtual machine that’s conceivably to much for a single script like that. > > [1] https://issues.apache.org/jira/browse/INFRA-15182 < > https://issues.apache.org/jira/browse/INFRA-15182> > > — > Denis > > > On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <dsetrak...@apache.org> > wrote: > > > > On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov <voze...@gridgain.com> > > wrote: > > > >> I am not sure it is good idea to send requests to 3rd-party addresses > from > >> Ignite node. Let's do not make the same mistakes again. > >> > > > > Agree with Vladimir. > > > > We obviously have CGI support on the website. Can someone explain why CGI > > is not possible to use? > > > > > >> > >> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <anovi...@gridgain.com> > >> wrote: > >> > >>> We may directly send request to GA from Ignite node: > >>> https://developers.google.com/analytics/devguides/ > >> collection/protocol/v1/ > >>> <https://developers.google.com/analytics/devguides/ > >> collection/protocol/v1/ > >>>> > >>> Latest version can be received from maven central: > >>> https://repo1.maven.org/maven2/org/apache/ignite/ > >>> ignite-core/maven-metadata.xml <https://repo1.maven.org/ > >>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml> > >>> > >>> > >>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <dsetrak...@apache.org> > >>> написал(а): > >>>> > >>>> Denis, > >>>> > >>>> I am not sure I understand. We already do have CGI enabled for > >>>> download.cgi. Is there something else we need? > >>>> > >>>> D. > >>>> > >>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dma...@gridgain.com> > >> wrote: > >>>> > >>>>> There is an obstacle. There is no way to execute the script using PHP > >> or > >>>>> similar sever side language and trigger GA as discussed earlier: > >>>>> https://issues.apache.org/jira/browse/INFRA-15182 > >>>>> > >>>>> How else can we tackle this? > >>>>> > >>>>> Denis > >>>>> > >>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan < > >>> dsetrak...@apache.org> > >>>>> wrote: > >>>>> > >>>>>> I think it is safe to assume at this point that everyone is in > >> general > >>>>>> agreement, since there are no active objections. > >>>>>> > >>>>>> I have filed a ticket for the 2.3 release. Let's try to make it > >> happen: > >>>>>> https://issues.apache.org/jira/browse/IGNITE-6305 > >>>>>> > >>>>>> D. > >>>>>> > >>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan < > >>>>> dsetrak...@apache.org > >>>>>> <javascript:;>> > >>>>>> wrote: > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani < > >> raul....@evosent.com > >>>>>> <javascript:;>> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Yeah, I guess that's doable as well and requires less management > >>>>> effort > >>>>>>>> than my suggestion. We could use events [1] to store payload data > >>>>> (e.g. > >>>>>>>> IP, > >>>>>>>> version, etc.) > >>>>>>> > >>>>>>> > >>>>>>> Yes, we could use events or some other similar API provided by GA. > >>>>>>> > >>>>>>> > >>>>>>>> What the download page CGI developed in? PHP? > >>>>>>>> > >>>>>>> > >>>>>>> To be honest, no clue. I guess someone in the community can figure > >> it > >>>>>> out: > >>>>>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html > >>>>>>> > >>>>>>> > >>>>>>>> However, I'm not sure whether storing this data in a 3rd party > >>>>> (Google) > >>>>>> is > >>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if > >> there's > >>>>>>>> doubt > >>>>>>>> in the PMC, it's better to ask legal@. > >>>>>>> > >>>>>>> > >>>>>>> I am not sure there is anything to ask about. The whole Ignite > >> website > >>>>> is > >>>>>>> GA enabled, and all we are doing is accessing a standard web page > >> from > >>>>>> the > >>>>>>> Ignite web site. The information gathered from GA is available only > >> to > >>>>>> the > >>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by this > >>>>>>> question. > >>>>>>> > >>>>>>> Even ASF website itself uses GA: https://www.apache.org/ > >>>>>>> foundation/policies/privacy.html > >>>>>>> > >>>>>>> > >>>>>>>> I think Cos said it's OK; maybe Roman can pitch in. > >>>>>>>> > >>>>>>> > >>>>>>> Sure, would be nice to hear from Roman as well. > >>>>>>> > >>>>>>> > >>>>>>>> Cheers. > >>>>>>>> > >>>>>>>> [1] > >>>>>>>> https://developers.google.com/analytics/devguides/collection > >>>>>>>> /analyticsjs/events > >>>>>>>> > >>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan < > >>>>>>>> dsetrak...@apache.org <javascript:;>> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Raul, > >>>>>>>>> > >>>>>>>>> Could point about Javascript, it will not work because it > executes > >>>>> in > >>>>>>>> the > >>>>>>>>> browser. This means we need a server-side script, like CGI we are > >>>>>> using > >>>>>>>> on > >>>>>>>>> our download page. > >>>>>>>>> > >>>>>>>>> How about this approach. We create something like > >> ignite-version.cgi > >>>>>>>> script > >>>>>>>>> which will invoke a call to GA and then return the latest > version. > >>>>>>>>> > >>>>>>>>> This should work, right? > >>>>>>>>> > >>>>>>>>> D. > >>>>>>>>> > >>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani < > >>>>> raul....@evosent.com > >>>>>> <javascript:;>> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hey Dmitriy and all > >>>>>>>>>> > >>>>>>>>>> Also, since we have GA enabled for the website, we can track how > >>>>>> many > >>>>>>>>> times > >>>>>>>>>>> this page was accessed, which will be equal to the number of > >>>>>> starts. > >>>>>>>>> This > >>>>>>>>>>> way, the counter information is tracked and monitored by the > >>>>>> Ignite > >>>>>>>>> PMC. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Unfortunately this won't work because GA is loaded via > Javascript > >>>>> on > >>>>>>>> the > >>>>>>>>>> browser, so Google will never receive the page hit. > >>>>>>>>>> > >>>>>>>>>> Given the constraints, the most viable solution is an HTTPS > >>>>> endpoint > >>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST > >>>>> request. > >>>>>>>> The > >>>>>>>>>> simplest thing is to write each request in a log file, along > with > >>>>>> the > >>>>>>>>>> timestamp, the version reported by the client, maybe the IP (not > >>>>>> sure > >>>>>>>>> about > >>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK if > >>>>> you > >>>>>>>> make > >>>>>>>>> it > >>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the node > >>>>>> creates > >>>>>>>> on > >>>>>>>>>> first startup or something. > >>>>>>>>>> > >>>>>>>>>> This endpoint would need some basic DDoS protection and > >>>>> blacklisting > >>>>>>>> to > >>>>>>>>>> prevent data spoofing. > >>>>>>>>>> > >>>>>>>>>> If we'll be implementing this endpoint anyway, then there's no > >>>>> point > >>>>>>>>>> placing another file on Git or elsewhere for reporting the > latest > >>>>>>>>> versions: > >>>>>>>>>> the endpoint itself can return them. > >>>>>>>>>> > >>>>>>>>>> WDYT? > >>>>>>>>>> > >>>>>>>>>> Cheers. > >>>>>>>>>> > >>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan < > >>>>>>>>> dsetrak...@apache.org <javascript:;>> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Cos, Raul, > >>>>>>>>>>> > >>>>>>>>>>> Thanks for the feedback. I completely agree about Maven Central > >>>>>>>> being a > >>>>>>>>>> 3rd > >>>>>>>>>>> party repo (did not think about that initially). All your > >>>>>>>> suggestions > >>>>>>>>>> make > >>>>>>>>>>> sense, but I would like to keep it as simple as possible, and > so > >>>>>> far > >>>>>>>>>>> everything suggested required GIT dependencies and extra work. > >>>>>>>>>>> > >>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the > Ignite > >>>>>>>>> website > >>>>>>>>>>> which will have only the latest version. Every time a node > >>>>> starts, > >>>>>>>> it > >>>>>>>>>>> receives the latest version from the page and suggests that > >>>>> users > >>>>>>>>> upgrade > >>>>>>>>>>> if needed. > >>>>>>>>>>> > >>>>>>>>>>> Also, since we have GA enabled for the website, we can track > how > >>>>>>>> many > >>>>>>>>>> times > >>>>>>>>>>> this page was accessed, which will be equal to the number of > >>>>>> starts. > >>>>>>>>> This > >>>>>>>>>>> way, the counter information is tracked and monitored by the > >>>>>> Ignite > >>>>>>>>> PMC. > >>>>>>>>>>> > >>>>>>>>>>> This approach looks pretty innocent to me and everything is > kept > >>>>>> and > >>>>>>>>>>> managed within Apache. > >>>>>>>>>>> > >>>>>>>>>>> Thoughts? > >>>>>>>>>>> > >>>>>>>>>>> D. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik < > >>>>>>>> c...@apache.org <javascript:;>> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> I agree with Raul. > >>>>>>>>>>>> - providing a ping-back address to a 3rd party might be frown > >>>>>>>> upon by > >>>>>>>>>>> some. > >>>>>>>>>>>> And might have a consequences like stats collection about > >>>>>> users' > >>>>>>>>>>>> infrastructure. > >>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any > >>>>> binary > >>>>>>>>> data: > >>>>>>>>>>>> everything is clear text and could be easily monitored by > >>>>> any > >>>>>> of > >>>>>>>>> the > >>>>>>>>>>>> network > >>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a > >>>>> bit > >>>>>> of > >>>>>>>>> the > >>>>>>>>>>>> release > >>>>>>>>>>>> discipline. > >>>>>>>>>>>> - the binary data download in the runtime is my main concern. > >>>>>>>> This is > >>>>>>>>>> the > >>>>>>>>>>>> vector of MMA. What if someone gains the control over the > >>>>>>>>> repository > >>>>>>>>>>> and > >>>>>>>>>>>> replaces the file with some malicious content. > >>>>>>>>>>>> > >>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make a > >>>>> call > >>>>>>>> to > >>>>>>>>> an > >>>>>>>>>>>> external script to check upon the atest software version > >>>>>> available > >>>>>>>>> for > >>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd party > >>>>>>>>> server, > >>>>>>>>>> I > >>>>>>>>>>>> believe the best way would be to put this script on ASF infra > >>>>>> and > >>>>>>>>> have > >>>>>>>>>>> the > >>>>>>>>>>>> "update checker" running in a completely controlled > >>>>> environment. > >>>>>>>>>>> Actually, > >>>>>>>>>>>> I > >>>>>>>>>>>> recall we had this very discussion during the Incubation; I > >>>>> can > >>>>>>>>>> probably > >>>>>>>>>>>> dig > >>>>>>>>>>>> out the corresponding thread. > >>>>>>>>>>>> > >>>>>>>>>>>> Thoughts? > >>>>>>>>>>>> Cok > >>>>>>>>>>>> > >>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote: > >>>>>>>>>>>>> Hey guys > >>>>>>>>>>>>> > >>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party > >>>>>>>>> (Sonatype). > >>>>>>>>>>> We > >>>>>>>>>>>>> don't know what kind of data analysis or intelligence > >>>>>> extraction > >>>>>>>>> they > >>>>>>>>>>>> run. > >>>>>>>>>>>>> > >>>>>>>>>>>>> If Ignite servers all over the world were hitting maven.org > >>>>>>>>>>> periodically > >>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a > >>>>> clear > >>>>>>>>>>> indication > >>>>>>>>>>>>> of who is running an Ignite server. > >>>>>>>>>>>>> > >>>>>>>>>>>>> They could reverse lookup the IP address and find out what > >>>>>>>>>> corporation > >>>>>>>>>>> it > >>>>>>>>>>>>> is. > >>>>>>>>>>>>> > >>>>>>>>>>>>> How about having Ignite check the ASF Git directly? > >>>>>>>>>>>>> > >>>>>>>>>>>>> We could use the Git ssh API, but that would require a new > >>>>>>>>>> dependency, > >>>>>>>>>>>>> which I advise against. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new Git > >>>>>>>> tags: > >>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git > >>>>>>>>>>>>> > >>>>>>>>>>>>> Another option is to place a txt file in the root of the > >>>>>> master > >>>>>>>>>> branch > >>>>>>>>>>>> (e.g > >>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for > >>>>> each > >>>>>>>> major > >>>>>>>>>>>> version > >>>>>>>>>>>>> line (1.x, 2.x). > >>>>>>>>>>>>> > >>>>>>>>>>>>> I would advocate this last option, but it requires somebody > >>>>>>>>>> remembering > >>>>>>>>>>>> to > >>>>>>>>>>>>> update the file with every release, unless we automate it > >>>>>> with a > >>>>>>>>>> Maven > >>>>>>>>>>>>> plugin. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hope that helps! > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dma...@apache.org > >>>>>> <javascript:;>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> I see nothing wrong with this approach. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think? Is > >>>>> it > >>>>>>>> good > >>>>>>>>>> to > >>>>>>>>>>>> go? > >>>>>>>>>>>>> > >>>>>>>>>>>>> — > >>>>>>>>>>>>> Denis > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan < > >>>>>>>>>>> dsetrak...@apache.org <javascript:;> > >>>>>>>>>>>>> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a ticket > >>>>> on > >>>>>>>> it? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan < > >>>>>>>>>>>> dsetrak...@apache.org <javascript:;>> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Igniters, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> There has been lots of talk of proposals about various > >>>>>> usage > >>>>>>>>>> metrics > >>>>>>>>>>>> for > >>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to resurrect > >>>>>> the > >>>>>>>>> topic > >>>>>>>>>>> and > >>>>>>>>>>>>>>> propose something very simple and non-intrusive. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 1. Update Checker > >>>>>>>>>>>>>>> The main purpose of the update checker is not to collect > >>>>>>>>> metrics, > >>>>>>>>>>> but > >>>>>>>>>>>> to > >>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing > >>>>>>>>> maven.org > >>>>>>>>>>> and > >>>>>>>>>>>>>>> getting the version out of the metadata file: > >>>>>>>>>>>>>>> http://repo2.maven.org/maven2/ > >>>>>> org/apache/ignite/ignite-core/ > >>>>>>>>>>>>>>> maven-metadata.xml > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This way we do not send any information anywhere and, at > >>>>>> the > >>>>>>>>> same > >>>>>>>>>>>> time, > >>>>>>>>>>>>>>> urge our users to download and start using the latest > >>>>>>>> version of > >>>>>>>>>>>> Ignite. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 2. Startup Counter > >>>>>>>>>>>>>>> This piece is optional, but we can also get an insight in > >>>>>> how > >>>>>>>>> many > >>>>>>>>>>>> times > >>>>>>>>>>>>> a > >>>>>>>>>>>>>>> certain Ignite release gets started. This is just a cool > >>>>>>>> metric > >>>>>>>>>> for > >>>>>>>>>>>> the > >>>>>>>>>>>>>>> community to gauge the project popularity. You can think > >>>>> of > >>>>>>>> it > >>>>>>>>> as > >>>>>>>>>>> of a > >>>>>>>>>>>>> page > >>>>>>>>>>>>>>> visit counter shown on many websites. We can even decide > >>>>> to > >>>>>>>>>> display > >>>>>>>>>>>> this > >>>>>>>>>>>>>>> counter on the Ignite website as well. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every > >>>>>>>> release, > >>>>>>>>>> e.g. > >>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1 byte. > >>>>>>>> Every > >>>>>>>>>> time > >>>>>>>>>>>> an > >>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the > >>>>>>>> background. > >>>>>>>>>>> Then > >>>>>>>>>>>> we > >>>>>>>>>>>>>>> will be able to view the number of the total downloads > >>>>> for > >>>>>>>> this > >>>>>>>>>> JAR > >>>>>>>>>>> in > >>>>>>>>>>>>>>> Maven Central, which is essentially the number of starts > >>>>> of > >>>>>>>>> Ignite > >>>>>>>>>>>> nodes. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> *Note that neither of the above suggestions require > >>>>> Ignite > >>>>>> to > >>>>>>>>> send > >>>>>>>>>>> or > >>>>>>>>>>>>>>> track any user information whatsoever.* > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this > >>>>>>>> approach. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> D. > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>> > >>> > >> > >
