Supported Versions
Hi folks, Looking into the documentation for version support on the website, there's some areas that are unclear to me. On the website it says "NOTE: Spot requires Spark 2.1.0 if you are using Spark 1.6 please upgrade your Spark version to 2.1.0." Is that 2.1.0 EXACTLY, or anything 2.1.0 or higher? It would be great to have a list of all the versions of the pre-reqs on the site that were tested on. HDFS. HIVE. IMPALA. KAFKA. SPARK (YARN). YARN. Zookeeper. Thoughts? Thank you! Cheers, -Vivienne
Re: Hive Partition Names for numbers (SPOT-19 and SPOT-239)
Hi Jeremy, My personal vote would be to align around the two-digit zero-padded strings. It's more standardized, and the consistency in format makes it easier to run scripts as needed. In that sense, it should be viable to run a script to change the numbers that are currently stored, and then stay consistent going forward. We've been running into issues with things like this in $DAY_JOB and it's definitely worth getting consistent sooner rather than later, IMO. Cheers, -Vivienne On Mon, Mar 30, 2020 at 12:47 PM Jeremy Nelson wrote: > Greetings, > > I noticed that SPOT-19 and SPOT-239 describe two aspects of the same > problem – that is – HIVE partition names are stored as numbers (ie, > y=2020/m=3/d=4) but some software (like the SPOT-ML) expects these > partition names to be stored as two-digit zero-padded strings (ie, > y=2020/m=03/d=04). > > We should identify which approach is preferable (break the schema, or break > the software?) and then harmonize all affected systems to do the same > thing, one way or the other. > > Thanks, > > Jeremy >
Re: [DISCUSS] Migrate spot-demo code from ONI repository
As a general rule, the more things are centralized and owned by the one project, I think the easier it will be to reliably make improvements and new development. On Mon, Mar 2, 2020 at 6:47 PM Tadd Wood wrote: > Similar to NFdump, I think we should start looking at migrating the > spot-demo code properly over from the original ONI repository so that it's > formally managed under the Apache Spot codebase. > > We should also look at revamping the demo code at some point soon, but I > think this would be a first step towards that effort. Thoughts? > > Thank you, > Tadd Wood >
Re: nfdump install instructions seem inaccurate
In the meantime, finally put in that PR to update the docs page to reflect the need for a different version of automake while the longer term plan is figured out. I don't do a lot of wordsmithing docs type things, so if anyone feels up to looking at this PR to make sure it's audience-friendly, that'd be great :) https://github.com/apache/incubator-spot/pull/162 Cheers, -Viv On Wed, Feb 5, 2020 at 12:25 PM Tadd Wood wrote: > I support moving/forking the code in until we decide how we want to proceed > with eliminating that dependency. > > Thank you, > Tadd Wood > > On Wed, Feb 5, 2020 at 7:55 AM Jeremy Nelson > wrote: > > > I agree with the sentiment that we should work towards eliminating the > > dependency. > > In the meantime, until we figure out how to do that, it may be necessary > > for us to make changes to spot-nfdump. > > I would suggest that just for our own sanity, we should include/fork the > > project (open-network-insight/spot-nfdump) so we can make commits to it, > > and so that unknown people can't make changes that might affect our > users. > > We could update the documentation to direct users to install our > maintained > > version, instead of the external copy. > > > > Jeremy > > > > > > On Wed, Feb 5, 2020 at 12:08 AM Nate Smith > wrote: > > > > > I’ve not been active in this project for a while, so forgive me if this > > > has already been addressed. > > > > > > I’m assuming this is for spot-nfdump? > > > > > > If so it’s worth saying unless someone has been maintaining it we > should > > > look at removing the dependency. > > > > > > Originally we cooked our own version due to time stamp issues. But this > > > should really be handled post ingestion IMHO. > > > > > > - nathanael > > > > > > > On Feb 4, 2020, at 3:40 PM, skip cruse wrote: > > > > > > > > I noticed when setting up nfdump for spot-ingest that there were > some > > > > errors around the version of automake that’s requried. Apparently > > > automake > > > > 1.14 is required and after installing that everything worked fine. I > > did > > > > some digging and it looks like this was already raised via SPOT-178: > > > > https://issues.apache.org/jira/browse/SPOT-178. Perhaps someone > could > > > > update the website to reflect this dependency and add the additional > > > setup > > > > steps in there as well? > > > > > > > > Cheers, > > > > Skip > > > > -- > > > > if( bool halfWayThere == true); > > > > printf "WAH! LIVIN ON A PRAYER"; > > > > > >
Re: [GitHub] [incubator-spot] yellingviv closed pull request #158: add Apache Foundation links on supporting apache page
Well that wasn't supposed to happen, I was just cleaning up my local drive, yikes, sorry folks! On Mon, Mar 2, 2020 at 3:20 PM GitBox wrote: > yellingviv closed pull request #158: add Apache Foundation links on > supporting apache page > URL: https://github.com/apache/incubator-spot/pull/158 > > > > > > This is an automated message from the Apache Git Service. > To respond to the message, please log on to GitHub and use the > URL above to go to the specific comment. > > For queries about this service, please contact Infrastructure at: > us...@infra.apache.org > > > With regards, > Apache Git Services >
Re: [asf-site] Whimsy report indicates missing Apache links
You got it! Gonna try to get on this today :) On Wed, Feb 19, 2020 at 5:56 AM Tadd Wood wrote: > Vivienne, can you also add a link for Security to the list? This link > should suffice: https://www.apache.org/security/ > <http://www.apache.org/security/> > > Thank you, > Tadd Wood > > On Thu, Jan 30, 2020 at 9:18 PM Tadd Wood wrote: > > > An extra hand would be greatly appreciated. I've already gone ahead and > > created a JIRA to track this: > > https://issues.apache.org/jira/browse/SPOT-299 > > > > Take a look and let me know when your changes are ready for review. > > > > Thank you, > > Tadd Wood > > > > On Wed, Jan 29, 2020 at 6:51 PM Vivienne Pustell > > wrote: > > > >> That sounds like a great idea, Tadd -- makes sense to get the links in > >> place so that everything is in order and connected where it should be. > Do > >> you have that under control, or would you like a hand? > >> > >> Best, > >> -Vivienne > >> > >> On Wed, Jan 29, 2020 at 6:37 PM Tadd Wood wrote: > >> > >> > I took a look into the Whimsy report for Spot website and it looks > like > >> > we’re missing a number of Apache foundation links: > >> > > >> > https://whimsy.apache.org/pods/project/spot > >> > > >> > I think we could easily add these links to this page to fix this > issue: > >> > > >> > https://spot.incubator.apache.org/get-started/supporting-apache/ > >> > > >> > Thoughts? > >> > > >> > Thank you, > >> > Tadd Wood > >> > > >> > > >
Re: nfdump install instructions seem inaccurate
I'm working on another website fix, I'll add this into my queue as well and get it all done together so I can get the PRs submitted soon. On Tue, Feb 4, 2020 at 7:25 PM Vivienne Pustell wrote: > Sounds like a solid way to keep the ball rolling on getting the docs > clearer and more up to date! > > On Tue, Feb 4, 2020 at 3:40 PM skip cruse wrote: > >> I noticed when setting up nfdump for spot-ingest that there were some >> errors around the version of automake that’s requried. Apparently automake >> 1.14 is required and after installing that everything worked fine. I did >> some digging and it looks like this was already raised via SPOT-178: >> https://issues.apache.org/jira/browse/SPOT-178. Perhaps someone could >> update the website to reflect this dependency and add the additional setup >> steps in there as well? >> >> Cheers, >> Skip >> -- >> if( bool halfWayThere == true); >> printf "WAH! LIVIN ON A PRAYER"; >> >
Re: [asf-site] Whimsy report indicates missing Apache links
That sounds like a great idea, Tadd -- makes sense to get the links in place so that everything is in order and connected where it should be. Do you have that under control, or would you like a hand? Best, -Vivienne On Wed, Jan 29, 2020 at 6:37 PM Tadd Wood wrote: > I took a look into the Whimsy report for Spot website and it looks like > we’re missing a number of Apache foundation links: > > https://whimsy.apache.org/pods/project/spot > > I think we could easily add these links to this page to fix this issue: > > https://spot.incubator.apache.org/get-started/supporting-apache/ > > Thoughts? > > Thank you, > Tadd Wood >
Re: [asf-site] Clean-up duplicate files and directories
I’m always a huge fan of cleaning up files, especially cleaning out old Wordpress stuff, yikes. A better baseline structure sounds like a really valuable use of effort. Best, -Vivienne On Thu, Jan 23, 2020 at 10:21 PM Tadd Wood wrote: > After digging into the asf-site branch for a bit, I've noticed there are > duplicate files and directories which could likely be removed. Some of > these even look like Wordpress carryovers that can either be removed or > consolidated. I’d like to clean this up so we have a better baseline for > the site's structure. I think in a follow-up project it would be worth > reorganizing the site a bit too so that it's easier to expand and add-to. > Thoughts? > > Thank you, > Tadd Wood >
