SPARK-5747 <https://issues.apache.org/jira/browse/SPARK-5747>: Review all Bash scripts for word splitting bugs
I’ll file sub-tasks under this issue. Feel free to pitch in people! Nick On Wed Feb 11 2015 at 3:07:51 PM Ted Yu <yuzhih...@gmail.com> wrote: > After some googling / trial and error, I got the following working > (against a directory with space in its name): > > #!/usr/bin/env bash > OLDIFS="$IFS" # save it > IFS="" # don't split on any white space > dir="$1/*" > for f in "$dir"; do > cat $f > done > IFS=$OLDIFS # restore IFS > > Cheers > > On Wed, Feb 11, 2015 at 2:47 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> The tragic thing here is that I was asked to review the patch that >> introduced this >> <https://github.com/apache/spark/pull/3377#issuecomment-68077315>, and >> totally missed it... :( >> >> On Wed Feb 11 2015 at 2:46:35 PM Nicholas Chammas < >> nicholas.cham...@gmail.com> wrote: >> >>> lol yeah, I changed the path for the email... turned out to be the issue >>> itself. >>> >>> >>> On Wed Feb 11 2015 at 2:43:09 PM Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>> I see. >>>> '/path/to/spark-1.2.1-bin-hadoop2.4' didn't contain space :-) >>>> >>>> On Wed, Feb 11, 2015 at 2:41 PM, Nicholas Chammas < >>>> nicholas.cham...@gmail.com> wrote: >>>> >>>>> Found it: >>>>> >>>>> https://github.com/apache/spark/compare/v1.2.0...v1.2.1#diff- >>>>> 73058f8e51951ec0b4cb3d48ade91a1fR73 >>>>> >>>>> GRRR BASH WORD SPLITTING >>>>> >>>>> My path has a space in it... >>>>> >>>>> Nick >>>>> >>>>> On Wed Feb 11 2015 at 2:37:39 PM Nicholas Chammas < >>>>> nicholas.cham...@gmail.com> wrote: >>>>> >>>>>> This is what get: >>>>>> >>>>>> spark-1.2.1-bin-hadoop2.4$ ls -1 lib/ >>>>>> datanucleus-api-jdo-3.2.6.jar >>>>>> datanucleus-core-3.2.10.jar >>>>>> datanucleus-rdbms-3.2.9.jar >>>>>> spark-1.2.1-yarn-shuffle.jar >>>>>> spark-assembly-1.2.1-hadoop2.4.0.jar >>>>>> spark-examples-1.2.1-hadoop2.4.0.jar >>>>>> >>>>>> So that looks correct… Hmm. >>>>>> >>>>>> Nick >>>>>> >>>>>> >>>>>> On Wed Feb 11 2015 at 2:34:51 PM Ted Yu <yuzhih...@gmail.com> wrote: >>>>>> >>>>>>> I downloaded 1.2.1 tar ball for hadoop 2.4 >>>>>>> I got: >>>>>>> >>>>>>> ls lib/ >>>>>>> datanucleus-api-jdo-3.2.6.jar datanucleus-rdbms-3.2.9.jar >>>>>>> spark-assembly-1.2.1-hadoop2.4.0.jar >>>>>>> datanucleus-core-3.2.10.jar spark-1.2.1-yarn-shuffle.jar >>>>>>> spark-examples-1.2.1-hadoop2.4.0.jar >>>>>>> >>>>>>> FYI >>>>>>> >>>>>>> On Wed, Feb 11, 2015 at 2:27 PM, Nicholas Chammas < >>>>>>> nicholas.cham...@gmail.com> wrote: >>>>>>> >>>>>>>> I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran >>>>>>>> sbin/start-all.sh >>>>>>>> on my OS X. >>>>>>>> >>>>>>>> Failed to find Spark assembly in /path/to/spark-1.2.1-bin-hadoo >>>>>>>> p2.4/lib >>>>>>>> You need to build Spark before running this program. >>>>>>>> >>>>>>>> Did the same for 1.2.0 and it worked fine. >>>>>>>> >>>>>>>> Nick >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>> >
