[jira] [Updated] (SPARK-22459) EdgeDirection "Either" Does Not Considerate Real "Either" Direction
[ https://issues.apache.org/jira/browse/SPARK-22459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom updated SPARK-22459: Description: When running functions involving [EdgeDirection|https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/EdgeDirection.scala], the _EdgeDirection.Either_ type does not truly perform the actual specified direction (_either_). For instance, when using _Pregel API_ with a simple graph as shown: !http://www.icodeguru.com/CPP/10book/books/book9/images/fig6_1.gif! With _EdgeDirection.Either_ you would guess that vertex #3 will send a message (when activated) to vertices 1, 2, and 4, but in reality it does not. This might be bypassed, but in an expansive and ineffective way. Tested with 2.2.0, 2.1.1 and 2.1.2 spark versions. was: When running functions involving [EdgeDirection|https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/EdgeDirection.scala], the _EdgeDirection.Either_ type does not truly perform the actual specified direction (_either_). For instance, when using _Pregel API_ with a simple graph as shown: !http://www.icodeguru.com/CPP/10book/books/book9/images/fig6_1.gif|thumbnail! With _EdgeDirection.Either_ you would guess that vertex #3 will send a message (when activated) to vertices 1, 2, and 4, but in reality it does not. This might be bypassed, but in an expansive and ineffective way. Tested with 2.2.0, 2.1.1 and 2.1.2 spark versions. > EdgeDirection "Either" Does Not Considerate Real "Either" Direction > --- > > Key: SPARK-22459 > URL: https://issues.apache.org/jira/browse/SPARK-22459 > Project: Spark > Issue Type: Bug > Components: GraphX >Affects Versions: 2.1.1, 2.1.2, 2.2.0 > Environment: Windows 7. Yarn cluster / client mode. Tested with > 2.2.0, 2.1.1 and 2.1.2 spark versions. >Reporter: Tom > > When running functions involving > [EdgeDirection|https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/EdgeDirection.scala], > the _EdgeDirection.Either_ type does not truly perform the actual specified > direction (_either_). > For instance, when using _Pregel API_ with a simple graph as shown: > !http://www.icodeguru.com/CPP/10book/books/book9/images/fig6_1.gif! > With _EdgeDirection.Either_ you would guess that vertex #3 will send a > message (when activated) to vertices 1, 2, and 4, but in reality it does not. > This might be bypassed, but in an expansive and ineffective way. > Tested with 2.2.0, 2.1.1 and 2.1.2 spark versions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-22459) EdgeDirection "Either" Does Not Considerate Real "Either" Direction
[ https://issues.apache.org/jira/browse/SPARK-22459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom updated SPARK-22459: Description: When running functions involving [EdgeDirection|https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/EdgeDirection.scala], the _EdgeDirection.Either_ type does not truly perform the actual specified direction (_either_). For instance, when using _Pregel API_ with a simple graph as shown: !http://www.icodeguru.com/CPP/10book/books/book9/images/fig6_1.gif|thumbnail! With _EdgeDirection.Either_ you would guess that vertex #3 will send a message (when activated) to vertices 1, 2, and 4, but in reality it does not. This might be bypassed, but in an expansive and ineffective way. Tested with 2.2.0, 2.1.1 and 2.1.2 spark versions. was: When running functions involving [EdgeDirection|https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/EdgeDirection.scala], the _EdgeDirection.Either_ type does not truly perform the actual specified direction (_either_). For instance, when using _Pregel API_ with a simple graph as shown: !http://www.icodeguru.com/CPP/10book/books/book9/images/fig6_1.gif! With _EdgeDirection.Either_ you would guess that vertex #3 will send a message (when activated) to vertices 1, 2, and 4, but in reality it does not. This might be bypassed, but in an expansive and ineffective way. Tested with 2.2.0, 2.1.1 and 2.1.2 spark versions. > EdgeDirection "Either" Does Not Considerate Real "Either" Direction > --- > > Key: SPARK-22459 > URL: https://issues.apache.org/jira/browse/SPARK-22459 > Project: Spark > Issue Type: Bug > Components: GraphX >Affects Versions: 2.1.1, 2.1.2, 2.2.0 > Environment: Windows 7. Yarn cluster / client mode. Tested with > 2.2.0, 2.1.1 and 2.1.2 spark versions. >Reporter: Tom > > When running functions involving > [EdgeDirection|https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/EdgeDirection.scala], > the _EdgeDirection.Either_ type does not truly perform the actual specified > direction (_either_). > For instance, when using _Pregel API_ with a simple graph as shown: > !http://www.icodeguru.com/CPP/10book/books/book9/images/fig6_1.gif|thumbnail! > With _EdgeDirection.Either_ you would guess that vertex #3 will send a > message (when activated) to vertices 1, 2, and 4, but in reality it does not. > This might be bypassed, but in an expansive and ineffective way. > Tested with 2.2.0, 2.1.1 and 2.1.2 spark versions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-22459) EdgeDirection "Either" Does Not Considerate Real "Either" Direction
[ https://issues.apache.org/jira/browse/SPARK-22459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom updated SPARK-22459: Description: When running functions involving [EdgeDirection|https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/EdgeDirection.scala], the _EdgeDirection.Either_ type does not truly perform the actual specified direction (_either_). For instance, when using _Pregel API_ with a simple graph as shown: !http://www.icodeguru.com/CPP/10book/books/book9/images/fig6_1.gif! With _EdgeDirection.Either_ you would guess that vertex #3 will send a message (when activated) to vertices 1, 2, and 4, but in reality it does not. This might be bypassed, but in an expansive and ineffective way. Tested with 2.2.0, 2.1.1 and 2.1.2 spark versions. was: When running functions involving [EdgeDirection|https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/EdgeDirection.scala], the _EdgeDirection.Either_ type does not truly perform the actual specified direction (_either_). For instance, when using _Pregel API_ with a simple graph as shown: !data:image/png;base64,iVBORw0KGgoNSUhEUgAAAPkAAADLCAMAAACbI8UEh1BMVEX9/f0AAAB1dXWFhYX4+Pjl5eV6enr39/fj4+Py8vLo6OgsLCyCgoLs7Ox8fHxTU1PKyso8PDxNTU0vLy/c3NwiIiJFRUVLS0vFxcVcXFzOzs41NTVSUlIYGBisrKy9vb1tbW1mZmYeHh6fn5+UlJQLCwuysrKMjIylpaWZmZlgYGASEhKECRTxAAAUhElEQVR4nN1diZqqOgy2LIq1ICAtssi+Ku//fBdwOY4UBEThu//MOAoUG5ImaZq2q9X/FwAIo8rxYOKK/BBA2KZR7AeBL8a5Bfs/AMHI83jv+34W5XmOv1jFr8CK96aalGCY8pdB+mUTYf5tMWBFG9kpCzBhVYxhCtXPrB/UdyIAHNlJeAxKpsXnPI7OMcseCSGbdwzcnu2kUE/+XsyyjNv7sh4miaOeezyzRQDHqqNm7hqW72/NledxepaJHnUJPYxUpAYxhpAHJcrCa2zkrJ4kdrb7RcU/BHCDwuaa3AWrXaSFm3VbOd4wic1ZTc2GMzV0ZMqJhcHyyYXDkH7SuxDZo0vuWtSci0EVCR6zqBT5hUs8NgvVa2fPLkMoppGAzdAWW0UaGAdCsnH28UfwdOQrXRfwkYYoJKx9JHstglLD2iCUflq7L8LwL7nUfQmfayh+JX3ro3d6H57Rwfioct8E9IP8bWsUYqJFfw9tfT1/K8qCSI5LteyYVdMeagjGmvqHBLi3ox6aW8hQ0NUgZkSE2F41gxninlns2cGbJnKFFGj5Im0b1o89ney1bz9pK3xQvX7lPKQusanDPXL7cgQf9cdDgpzq9jTVvIhOC+zBuMjvJbM1zuRh1Y1T/9a7NcPo/VU/BjTJAHur2ObNbeHPh96iUl5sn7ZDa/ZtpOGlheVw2+SpsFfd6ztFN1td+SZ2pu0Nr9tXAQO6iwWMWFbZpq+OD1emg4hklFbOt5h3/kz8hVk2bNNZt/aD4ICaypv3UX1sLasU92SdtTVnrB4WJu4GEmkKGvA7QVhz6NJgVK7l1b9UE5vs3fmkVetlaFk6DkSow9JuZb1hjPBRrP7FtEYice2Up4RdVJ8NmqjDpwbnpkivTbPUiPzmKvR/wXtEbuuxKkTubzx/gPVB7Wp+udrgOfQrOYCmTZMVy242j/spTZ+3oe/+1qxUPB0qV4gODcr5fSUH2yPV/VkfWxkL2a529X1g3WYNvH60OAt1UQ7yJn1Cpipliz5SFaNkHlqNfMyYhrCarefiVrFwZG9S7zqEYGmdrS81GjXlxar3IR1EGg3wcmqNTJ0ZJrlkMcuyG/YZm9cDj8N/P2wopxp3aj27sZkbCt2snDGsyl1coJwrex9eSaKc0a6Hl/YHGdfjEEdmbpDDvmrCW72TctrDyEjFc5Nqo2BwbEr79QuEDcPYMptxIseJ1R93f7m+eTpw/aUcq47e3v09frv8dlb8c1a8nj3UZGuXzChVHag01WGglRX2VWwJBjrNGkJfbmvnGDEBXkPIl/pBACu+RPkCeFB6Tav6FQig/OUBqE+Uf7dj5Uv9Qag+VB9X1VX3O6yuZerj5UtdqDoL/p4FHsMc2HwNbrwAWFcHjoJIgV8WFzYUTQ3KB9LW/VlhMq9uN3TxblVr6iEbDgwZbI/7siSfUfu2MDg8Pcg/7chCTd/g23iugIBfhDstWjgBbs3zVQ2khKv+xdRQAwxaNZxHKCpgTlgtvUclOOMS6eWlOQPROVf/PZsWtdwdWx211Nkvym8vFQ/VDYU+cTQb2Yh7oY9n664KWJvNzsyKzwly6fSVxnBhQXeeZagxcyvecGKU5a+MtdRbbyynxC2lgJC6uw9ePQGgkJnd9iYsctnRPJbKuDRR9tFumk3RmiENYLmpQY3Hlpp0aTGZFbiQPgMlNyja3WDDPXX4ANDvZenURjUvoqI/N0AcPobDPdQ/BAk5euhnXuCg/0iDYf8zyoKI4r6PzFCHhLZ/BsvWO4fOn640n4cQLV3rSc72tLjIaw0+dsxegSLoh+cn6RBiTe71yCDnyAszaTdgk8Q9SIcx+su58kn0GRkXIqK2JNnMDmweuLcOFh8T9YXFyom0jj8/hEPgQtJneP7HgEZaddks9q22AoZ2bBgmQ9eydwreUENOmnX4vPHlQLI8Vq/HDVbYJ2KnwPPuEVHst3VyLp2JjoKrE263rLwBKQ9QUsS3SmGZ+B3upZQRmzpabhyKY8PF/Qfo2uZmvSDCBSvb6AzDJP7Vy15VyTLEj1vYBz0ZXVrSBPCGELEl0REoAdIWlQAKYnKNxvn4X5UFN2DUPSV6wBuXkrjW+kNXLTS/EaQFAtjFKLGjRY2srLxrKjb5Y5OAFZ0cPVKe1REPjXyDSNCZ7af4JNQy5Q+J/DaND6EmGssifIVrnheNJLh1bDPMSTy7blr+pjkXHAsGsdYbvS94HGIcOXY9pYLhpRHHOqVIzTm0QIXg1SynjP7xZcv8E50OD5tOft8ADFO7ytH1lUk2F/MsrZYVhlntxLp69K4WxO7eN+WDXvIsMSOjb39MslL2UhYq6lEM3zXqPPllcRxuSlZqTOi2XQAEWGKXE0rAqfPGcGuphFHT7U6CdYb/HR/XeRJg0XE2FnbfZqxaqHXYoB0RKaKa5H9kj6d80icmGDqj57DPPS1tRJzYu/bgwV8Mvs30kDIUXpoDpFRYqH1IuBWGXUfcF0f5LitI70CKMkbaFa1+XAujHFiyc/R6X25pY3iu1rZyWZTDsx129UpeUWq44S63ZR8WRzk8O2g/hJRRlJftfGHSDhSTmMNiQsoYDacsTcMJoubsB3YeRtnzpWk4KQvVd5OTGsDaYYS0XzXcaiGUl8ZsRJL1JxruhfTBt5kGA43ZA6UnM3z486bhFkE5FAmiTLR9j5Ea7nRtVfNTbm2QPi7Ob2mnMRrutAyeC6X3Eowc2flIw80NKS69l7EpSRYyhxPxkPZ5gc1Q9UYHhJQxGq6U9vl5DoyDo3njy4/rq/2K5x26A0QobIbBBwCPiUwY9uw8FyKinT8arh/lyczfzrcbcvwwQWOUPS89mVl5zhtmr0H9TljaqHY+whROB5jbIftxfu2oOTe3ONwTfunHbFnSXBJiOJQBaV8PWGPcn6mA/eLgTpCKhO1Run22aWm8qxbBJJnko6LO3e38m4IPzwRl0zz1UZ7MbDyXfMd+v9BJP1hjpF2hh6q/reUEz3T0jjWQhmHcGMssfTUpQoS1Jks9GyXt1hz2fMcmagSpaemjMEra52jnilz0WgqoN0ppHxWZ+DHlIFVD/10+yzCMk/ZfezIwVd+u2jUUo8bPDfVXy6fUipzHG3JKp84gHyXtnjoibHnDcKPEe6oTvE7B+xzjPBn9h+18d7Yd/wszH0eNnyv67xYLMoLCPn9j4qPVI5jYEFDvR7odrIBnJ+oUPbMmjDE+nHUc384HYReHZOxIwjuM6p+XPP8J5copIe63JgyMs2r6L/x2wZMZdaqeWRMKGiG4P6EcRkXyzcWDR40o/oJyzIZkj7/Y8S2lfUQEsocP91mdBUVm7I4JJBNgrIb7rlWDqZ0cvrxMtqXpy7Nq24wknwfU32CchvvuCnjrS4LEr5vNUd5rP2kfJ6w8NExG9hqLfkyOURruQ2nvJApGhOk3B/hDjOurvUj7hPzZZU6x/42DiEbw/Fu6HfCWX6Bs95Np7OOiUV/S7YKnM+rkwZcWjMyTGaEW3wEAHCHmd5PYR42ffycOt2WTxKRvg/ENjBo/
[jira] [Created] (SPARK-22459) EdgeDirection "Either" Does Not Considerate real "either" direction
Tom created SPARK-22459: --- Summary: EdgeDirection "Either" Does Not Considerate real "either" direction Key: SPARK-22459 URL: https://issues.apache.org/jira/browse/SPARK-22459 Project: Spark Issue Type: Bug Components: GraphX Affects Versions: 2.2.0, 2.1.2, 2.1.1 Environment: Windows 7. Yarn cluster / client mode. Tested with 2.2.0, 2.1.1 and 2.1.2 spark versions. Reporter: Tom When running functions involving [EdgeDirection|https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/EdgeDirection.scala], the _EdgeDirection.Either_ type does not truly perform the actual specified direction (_either_). For instance, when using _Pregel API_ with a simple graph as shown: !data:image/png;base64,iVBORw0KGgoNSUhEUgAAAPkAAADLCAMAAACbI8UEh1BMVEX9/f0AAAB1dXWFhYX4+Pjl5eV6enr39/fj4+Py8vLo6OgsLCyCgoLs7Ox8fHxTU1PKyso8PDxNTU0vLy/c3NwiIiJFRUVLS0vFxcVcXFzOzs41NTVSUlIYGBisrKy9vb1tbW1mZmYeHh6fn5+UlJQLCwuysrKMjIylpaWZmZlgYGASEhKECRTxAAAUhElEQVR4nN1diZqqOgy2LIq1ICAtssi+Ku//fBdwOY4UBEThu//MOAoUG5ImaZq2q9X/FwAIo8rxYOKK/BBA2KZR7AeBL8a5Bfs/AMHI83jv+34W5XmOv1jFr8CK96aalGCY8pdB+mUTYf5tMWBFG9kpCzBhVYxhCtXPrB/UdyIAHNlJeAxKpsXnPI7OMcseCSGbdwzcnu2kUE/+XsyyjNv7sh4miaOeezyzRQDHqqNm7hqW72/NledxepaJHnUJPYxUpAYxhpAHJcrCa2zkrJ4kdrb7RcU/BHCDwuaa3AWrXaSFm3VbOd4wic1ZTc2GMzV0ZMqJhcHyyYXDkH7SuxDZo0vuWtSci0EVCR6zqBT5hUs8NgvVa2fPLkMoppGAzdAWW0UaGAdCsnH28UfwdOQrXRfwkYYoJKx9JHstglLD2iCUflq7L8LwL7nUfQmfayh+JX3ro3d6H57Rwfioct8E9IP8bWsUYqJFfw9tfT1/K8qCSI5LteyYVdMeagjGmvqHBLi3ox6aW8hQ0NUgZkSE2F41gxninlns2cGbJnKFFGj5Im0b1o89ney1bz9pK3xQvX7lPKQusanDPXL7cgQf9cdDgpzq9jTVvIhOC+zBuMjvJbM1zuRh1Y1T/9a7NcPo/VU/BjTJAHur2ObNbeHPh96iUl5sn7ZDa/ZtpOGlheVw2+SpsFfd6ztFN1td+SZ2pu0Nr9tXAQO6iwWMWFbZpq+OD1emg4hklFbOt5h3/kz8hVk2bNNZt/aD4ICaypv3UX1sLasU92SdtTVnrB4WJu4GEmkKGvA7QVhz6NJgVK7l1b9UE5vs3fmkVetlaFk6DkSow9JuZb1hjPBRrP7FtEYice2Up4RdVJ8NmqjDpwbnpkivTbPUiPzmKvR/wXtEbuuxKkTubzx/gPVB7Wp+udrgOfQrOYCmTZMVy242j/spTZ+3oe/+1qxUPB0qV4gODcr5fSUH2yPV/VkfWxkL2a529X1g3WYNvH60OAt1UQ7yJn1Cpipliz5SFaNkHlqNfMyYhrCarefiVrFwZG9S7zqEYGmdrS81GjXlxar3IR1EGg3wcmqNTJ0ZJrlkMcuyG/YZm9cDj8N/P2wopxp3aj27sZkbCt2snDGsyl1coJwrex9eSaKc0a6Hl/YHGdfjEEdmbpDDvmrCW72TctrDyEjFc5Nqo2BwbEr79QuEDcPYMptxIseJ1R93f7m+eTpw/aUcq47e3v09frv8dlb8c1a8nj3UZGuXzChVHag01WGglRX2VWwJBjrNGkJfbmvnGDEBXkPIl/pBACu+RPkCeFB6Tav6FQig/OUBqE+Uf7dj5Uv9Qag+VB9X1VX3O6yuZerj5UtdqDoL/p4FHsMc2HwNbrwAWFcHjoJIgV8WFzYUTQ3KB9LW/VlhMq9uN3TxblVr6iEbDgwZbI/7siSfUfu2MDg8Pcg/7chCTd/g23iugIBfhDstWjgBbs3zVQ2khKv+xdRQAwxaNZxHKCpgTlgtvUclOOMS6eWlOQPROVf/PZsWtdwdWx211Nkvym8vFQ/VDYU+cTQb2Yh7oY9n664KWJvNzsyKzwly6fSVxnBhQXeeZagxcyvecGKU5a+MtdRbbyynxC2lgJC6uw9ePQGgkJnd9iYsctnRPJbKuDRR9tFumk3RmiENYLmpQY3Hlpp0aTGZFbiQPgMlNyja3WDDPXX4ANDvZenURjUvoqI/N0AcPobDPdQ/BAk5euhnXuCg/0iDYf8zyoKI4r6PzFCHhLZ/BsvWO4fOn640n4cQLV3rSc72tLjIaw0+dsxegSLoh+cn6RBiTe71yCDnyAszaTdgk8Q9SIcx+su58kn0GRkXIqK2JNnMDmweuLcOFh8T9YXFyom0jj8/hEPgQtJneP7HgEZaddks9q22AoZ2bBgmQ9eydwreUENOmnX4vPHlQLI8Vq/HDVbYJ2KnwPPuEVHst3VyLp2JjoKrE263rLwBKQ9QUsS3SmGZ+B3upZQRmzpabhyKY8PF/Qfo2uZmvSDCBSvb6AzDJP7Vy15VyTLEj1vYBz0ZXVrSBPCGELEl0REoAdIWlQAKYnKNxvn4X5UFN2DUPSV6wBuXkrjW+kNXLTS/EaQFAtjFKLGjRY2srLxrKjb5Y5OAFZ0cPVKe1REPjXyDSNCZ7af4JNQy5Q+J/DaND6EmGssifIVrnheNJLh1bDPMSTy7blr+pjkXHAsGsdYbvS94HGIcOXY9pYLhpRHHOqVIzTm0QIXg1SynjP7xZcv8E50OD5tOft8ADFO7ytH1lUk2F/MsrZYVhlntxLp69K4WxO7eN+WDXvIsMSOjb39MslL2UhYq6lEM3zXqPPllcRxuSlZqTOi2XQAEWGKXE0rAqfPGcGuphFHT7U6CdYb/HR/XeRJg0XE2FnbfZqxaqHXYoB0RKaKa5H9kj6d80icmGDqj57DPPS1tRJzYu/bgwV8Mvs30kDIUXpoDpFRYqH1IuBWGXUfcF0f5LitI70CKMkbaFa1+XAujHFiyc/R6X25pY3iu1rZyWZTDsx129UpeUWq44S63ZR8WRzk8O2g/hJRRlJftfGHSDhSTmMNiQsoYDacsTcMJoubsB3YeRtnzpWk4KQvVd5OTGsDaYYS0XzXcaiGUl8ZsRJL1JxruhfTBt5kGA43ZA6UnM3z486bhFkE5FAmiTLR9j5Ea7nRtVfNTbm2QPi7Ob2mnMRrutAyeC6X3Eowc2flIw80NKS69l7EpSRYyhxPxkPZ5gc1Q9UYHhJQxGq6U9vl5DoyDo3njy4/rq/2K5x26A0QobIbBBwCPiUwY9uw8FyKinT8arh/lyczfzrcbcvwwQWOUPS89mVl5zhtmr0H9TljaqHY+whROB5jbIftxfu2oOTe3ONwTfunHbFnSXBJiOJQBaV8PWGPcn6mA/eLgTpCKhO1Run22aWm8qxbBJJnko6LO3e38m4IPzwRl0zz1UZ7MbDyXfMd+v9BJP1hjpF2hh6q/reUEz3T0jjWQhmHcGMssfTUpQoS1Jks9GyXt1hz2fMcmagSpaemjMEra52jnilz0WgqoN0ppHxWZ+DHlIFVD/10+yzCMk/ZfezIwVd+u2jUUo8bPDfVXy6fUipzHG3JKp84gHyXtnjoibHnDcKPEe6oTvE7B+xzjPBn9h+18d7Yd/wszH0eNnyv67xYLMoLCPn9j4qPVI5jYEFDvR7odrIBnJ+oUPbMmjDE+nHUc384HYReHZOxIwjuM6p+XPP8J5copIe63JgyMs2r6L/x2wZMZdaqeWRMKGiG4P6EcRkXyzcWDR40o/oJyzIZkj7/Y8S2lfUQEsocP91mdBUVm7I4JJBNgrIb7rlWDqZ0cvrxMtqXpy7Nq24wknwfU32CchvvuCnjrS4LEr5vNUd5rP2kfJ6w8NExG9hqLfkyOURruQ2nvJApGhOk3B/hDjOurvUj7hPzZZU6x/42DiEbw/Fu6HfCWX6Bs95Np7OOiUV/S7YKnM+rkwZcWjMyTGaEW3wEAHCHmd5PYR42ffycOt2WTxKRvg/ENjBo/974xxmKYBcl+uKvNKO9V0SfPEAKG5qg/ndg7yp4rqjyxtEsxSnqNmU1nQJfhvW43ReJ/s0tKwSjKp7bnipyQ+NcLy42i3NKntOdCZCdy+vOZvePmq9kTtvNtFibBDFtWKWicVZuK57xyKtB5jgVZRkYgp7JqwNAT9f3WQN/AKMonG10CLnHMH+v0O0b1WKaSduBp4evqmz/DuIzfaaLOOEDkPFvS9CSRiVEA2Eyai1T9DrNFJoRUJ7922/5gXGTic09GEp0wm3WJ1HFZYR/3WCzfUXuvi/AdGCOl/SMNBzzdORozLxo5R/9cYkkydY7XcIzScJ95MjBznPY9aX+G33sy601hD54m/
[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList
[
https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114179#comment-16114179
]
Tom commented on SPARK-21402:
-
Additional comment when there are multiple datatypes which are not easily
convert to one other (e.g double <-> string) the system get crazy (as well the
instruction in the error message didn't help) -
{code:bash}
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x00011be827bc, pid=992, tid=0x1c03
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build
1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode bsd-amd64
compressed oops)
# Problematic frame:
# v ~StubRoutines::jlong_disjoint_arraycopy
#
# Failed to write core dump. Core dumps have been disabled. To enable core
dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
{code}
> Java encoders - switch fields on collectAsList
> --
>
> Key: SPARK-21402
> URL: https://issues.apache.org/jira/browse/SPARK-21402
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.1
> Environment: mac os
> spark 2.1.1
> Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_121
>Reporter: Tom
>
> I have the following schema in a dataset -
> root
> |-- userId: string (nullable = true)
> |-- data: map (nullable = true)
> ||-- key: string
> ||-- value: struct (valueContainsNull = true)
> |||-- startTime: long (nullable = true)
> |||-- endTime: long (nullable = true)
> |-- offset: long (nullable = true)
> And I have the following classes (+ setter and getters which I omitted for
> simplicity) -
>
> {code:java}
> public class MyClass {
> private String userId;
> private Map data;
> private Long offset;
> }
> public class MyDTO {
> private long startTime;
> private long endTime;
> }
> {code}
> I collect the result the following way -
> {code:java}
> Encoder myClassEncoder = Encoders.bean(MyClass.class);
> Dataset results = raw_df.as(myClassEncoder);
> List lst = results.collectAsList();
> {code}
>
> I do several calculations to get the result I want and the result is correct
> all through the way before I collect it.
> This is the result for -
> {code:java}
> results.select(results.col("data").getField("2017-07-01").getField("startTime")).show(false);
> {code}
> |data[2017-07-01].startTime|data[2017-07-01].endTime|
> +-+--+
> |1498854000|1498870800 |
> This is the result after collecting the reuslts for -
> {code:java}
> MyClass userData = results.collectAsList().get(0);
> MyDTO userDTO = userData.getData().get("2017-07-01");
> System.out.println("userDTO startTime: " + userDTO.getStartTime());
> System.out.println("userDTO endTime: " + userDTO.getEndTime());
> {code}
> --
> data startTime: 1498870800
> data endTime: 1498854000
> I tend to believe it is a spark issue. Would love any suggestions on how to
> bypass it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21402) Java encoders - switch fields on collectAsList
[
https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom updated SPARK-21402:
Priority: Major (was: Minor)
> Java encoders - switch fields on collectAsList
> --
>
> Key: SPARK-21402
> URL: https://issues.apache.org/jira/browse/SPARK-21402
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.1
> Environment: mac os
> spark 2.1.1
> Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_121
>Reporter: Tom
>
> I have the following schema in a dataset -
> root
> |-- userId: string (nullable = true)
> |-- data: map (nullable = true)
> ||-- key: string
> ||-- value: struct (valueContainsNull = true)
> |||-- startTime: long (nullable = true)
> |||-- endTime: long (nullable = true)
> |-- offset: long (nullable = true)
> And I have the following classes (+ setter and getters which I omitted for
> simplicity) -
>
> {code:java}
> public class MyClass {
> private String userId;
> private Map data;
> private Long offset;
> }
> public class MyDTO {
> private long startTime;
> private long endTime;
> }
> {code}
> I collect the result the following way -
> {code:java}
> Encoder myClassEncoder = Encoders.bean(MyClass.class);
> Dataset results = raw_df.as(myClassEncoder);
> List lst = results.collectAsList();
> {code}
>
> I do several calculations to get the result I want and the result is correct
> all through the way before I collect it.
> This is the result for -
> {code:java}
> results.select(results.col("data").getField("2017-07-01").getField("startTime")).show(false);
> {code}
> |data[2017-07-01].startTime|data[2017-07-01].endTime|
> +-+--+
> |1498854000|1498870800 |
> This is the result after collecting the reuslts for -
> {code:java}
> MyClass userData = results.collectAsList().get(0);
> MyDTO userDTO = userData.getData().get("2017-07-01");
> System.out.println("userDTO startTime: " + userDTO.getStartTime());
> System.out.println("userDTO endTime: " + userDTO.getEndTime());
> {code}
> --
> data startTime: 1498870800
> data endTime: 1498854000
> I tend to believe it is a spark issue. Would love any suggestions on how to
> bypass it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList
[
https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096162#comment-16096162
]
Tom commented on SPARK-21402:
-
I have a small insight regarding this possible issue as I bumped into something
the appeared similar to me.
If the class we are creating instances for has a method getX when X is not a
member of this class it seems that something get messed up.
E.g. - I had another class with data member which is Long and static data
member of type Double, I have method of the following -
{code:java}
public double getX() {
return longDataMember / staticDouble;
}
{code}
This alone was enough to mess my class and to assign the value of 103079215144
to all the longDataMember.
> Java encoders - switch fields on collectAsList
> --
>
> Key: SPARK-21402
> URL: https://issues.apache.org/jira/browse/SPARK-21402
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.1
> Environment: mac os
> spark 2.1.1
> Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_121
>Reporter: Tom
>Priority: Minor
>
> I have the following schema in a dataset -
> root
> |-- userId: string (nullable = true)
> |-- data: map (nullable = true)
> ||-- key: string
> ||-- value: struct (valueContainsNull = true)
> |||-- startTime: long (nullable = true)
> |||-- endTime: long (nullable = true)
> |-- offset: long (nullable = true)
> And I have the following classes (+ setter and getters which I omitted for
> simplicity) -
>
> {code:java}
> public class MyClass {
> private String userId;
> private Map data;
> private Long offset;
> }
> public class MyDTO {
> private long startTime;
> private long endTime;
> }
> {code}
> I collect the result the following way -
> {code:java}
> Encoder myClassEncoder = Encoders.bean(MyClass.class);
> Dataset results = raw_df.as(myClassEncoder);
> List lst = results.collectAsList();
> {code}
>
> I do several calculations to get the result I want and the result is correct
> all through the way before I collect it.
> This is the result for -
> {code:java}
> results.select(results.col("data").getField("2017-07-01").getField("startTime")).show(false);
> {code}
> |data[2017-07-01].startTime|data[2017-07-01].endTime|
> +-+--+
> |1498854000|1498870800 |
> This is the result after collecting the reuslts for -
> {code:java}
> MyClass userData = results.collectAsList().get(0);
> MyDTO userDTO = userData.getData().get("2017-07-01");
> System.out.println("userDTO startTime: " + userDTO.getStartTime());
> System.out.println("userDTO endTime: " + userDTO.getEndTime());
> {code}
> --
> data startTime: 1498870800
> data endTime: 1498854000
> I tend to believe it is a spark issue. Would love any suggestions on how to
> bypass it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList
[
https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086179#comment-16086179
]
Tom commented on SPARK-21402:
-
Sorry, I was trying to hide my original code so it will be more understandable
(changed it now). It reads the class it is suppose to read but when collecting
it the fields are mixed.
> Java encoders - switch fields on collectAsList
> --
>
> Key: SPARK-21402
> URL: https://issues.apache.org/jira/browse/SPARK-21402
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.1
> Environment: mac os
> spark 2.1.1
> Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_121
>Reporter: Tom
>Priority: Minor
>
> I have the following schema in a dataset -
> root
> |-- userId: string (nullable = true)
> |-- data: map (nullable = true)
> ||-- key: string
> ||-- value: struct (valueContainsNull = true)
> |||-- startTime: long (nullable = true)
> |||-- endTime: long (nullable = true)
> |-- offset: long (nullable = true)
> And I have the following classes (+ setter and getters which I omitted for
> simplicity) -
>
> {code:java}
> public class MyClass {
> private String userId;
> private Map data;
> private Long offset;
> }
> public class MyDTO {
> private long startTime;
> private long endTime;
> }
> {code}
> I collect the result the following way -
> {code:java}
> Encoder myClassEncoder = Encoders.bean(MyClass.class);
> Dataset results = raw_df.as(myClassEncoder);
> List lst = results.collectAsList();
> {code}
>
> I do several calculations to get the result I want and the result is correct
> all through the way before I collect it.
> This is the result for -
> {code:java}
> results.select(results.col("data").getField("2017-07-01").getField("startTime")).show(false);
> {code}
> |data[2017-07-01].startTime|data[2017-07-01].endTime|
> +-+--+
> |1498854000|1498870800 |
> This is the result after collecting the reuslts for -
> {code:java}
> MyClass userData = results.collectAsList().get(0);
> MyDTO userDTO = userData.getData().get("2017-07-01");
> System.out.println("userDTO startTime: " + userDTO.getStartTime());
> System.out.println("userDTO endTime: " + userDTO.getEndTime());
> {code}
> --
> data startTime: 1498870800
> data endTime: 1498854000
> I tend to believe it is a spark issue. Would love any suggestions on how to
> bypass it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21402) Java encoders - switch fields on collectAsList
[
https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom updated SPARK-21402:
Description:
I have the following schema in a dataset -
root
|-- userId: string (nullable = true)
|-- data: map (nullable = true)
||-- key: string
||-- value: struct (valueContainsNull = true)
|||-- startTime: long (nullable = true)
|||-- endTime: long (nullable = true)
|-- offset: long (nullable = true)
And I have the following classes (+ setter and getters which I omitted for
simplicity) -
{code:java}
public class MyClass {
private String userId;
private Map data;
private Long offset;
}
public class MyDTO {
private long startTime;
private long endTime;
}
{code}
I collect the result the following way -
{code:java}
Encoder myClassEncoder = Encoders.bean(MyClass.class);
Dataset results = raw_df.as(myClassEncoder);
List lst = results.collectAsList();
{code}
I do several calculations to get the result I want and the result is correct
all through the way before I collect it.
This is the result for -
{code:java}
results.select(results.col("data").getField("2017-07-01").getField("startTime")).show(false);
{code}
|data[2017-07-01].startTime|data[2017-07-01].endTime|
+-+--+
|1498854000|1498870800 |
This is the result after collecting the reuslts for -
{code:java}
MyClass userData = results.collectAsList().get(0);
MyDTO userDTO = userData.getData().get("2017-07-01");
System.out.println("userDTO startTime: " + userDTO.getStartTime());
System.out.println("userDTO endTime: " + userDTO.getEndTime());
{code}
--
data startTime: 1498870800
data endTime: 1498854000
I tend to believe it is a spark issue. Would love any suggestions on how to
bypass it.
was:
I have the following schema in a dataset -
root
|-- userId: string (nullable = true)
|-- data: map (nullable = true)
||-- key: string
||-- value: struct (valueContainsNull = true)
|||-- startTime: long (nullable = true)
|||-- endTime: long (nullable = true)
|-- offset: long (nullable = true)
And I have the following classes (+ setter and getters which I omitted for
simplicity) -
{code:java}
public class MyClass {
private String userId;
private Map data;
private Long offset;
}
public class MyDTO {
private long startTime;
private long endTime;
}
{code}
I collect the result the following way -
{code:java}
Encoder myClassEncoder = Encoders.bean(MyClass.class);
Dataset results = raw_df.as(myClassEncoder);
List lst = results.collectAsList();
{code}
I do several calculations to get the result I want and the result is correct
all through the way before I collect it.
This is the result for -
{code:java}
results.select(results.col("data").getField("2017-07-01").getField("startTime")).show(false);
{code}
|data[2017-07-01].startTime|data[2017-07-01].endTime|
+-+--+
|1498854000|1498870800 |
This is the result after collecting the reuslts for -
{code:java}
MyClass userData = results.collectAsList().get(0);
MyDTO userDTO = userData.getData().get("2017-07-01");
System.out.println("userDTO startTime: " + userDTO.getSleepStartTime());
System.out.println("userDTO endTime: " + userDTO.getSleepEndTime());
{code}
--
data startTime: 1498870800
data endTime: 1498854000
I tend to believe it is a spark issue. Would love any suggestions on how to
bypass it.
> Java encoders - switch fields on collectAsList
> --
>
> Key: SPARK-21402
> URL: https://issues.apache.org/jira/browse/SPARK-21402
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.1
> Environment: mac os
> spark 2.1.1
> Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_121
>Reporter: Tom
>Priority: Minor
>
> I have the following schema in a dataset -
> root
> |-- userId: string (nullable = true)
> |-- data: map (nullable = true)
> ||-- key: string
> ||-- value: struct (valueContainsNull = true)
> |||-- startTime: long (nullable = true)
> |||-- endTime: long (nullable = true)
> |-- offset: long (nullable = true)
> And I have the following classes (+ setter and getters which I omitted for
> simplicity) -
>
> {code:java}
> public class MyClass {
> private String userId;
> private Map data;
> private Long offset;
> }
> public class MyDTO {
> private long startTime;
> private long endTime;
> }
> {code}
> I collect the result the following way -
> {code:java}
> Encoder myClassEncoder = Encoders.bean(MyClass.class);
> Dataset results = r
[jira] [Updated] (SPARK-21402) Java encoders - switch fields on collectAsList
[
https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom updated SPARK-21402:
Description:
I have the following schema in a dataset -
root
|-- userId: string (nullable = true)
|-- data: map (nullable = true)
||-- key: string
||-- value: struct (valueContainsNull = true)
|||-- startTime: long (nullable = true)
|||-- endTime: long (nullable = true)
|-- offset: long (nullable = true)
And I have the following classes (+ setter and getters which I omitted for
simplicity) -
{code:java}
public class MyClass {
private String userId;
private Map data;
private Long offset;
}
public class MyDTO {
private long startTime;
private long endTime;
}
{code}
I collect the result the following way -
{code:java}
Encoder myClassEncoder = Encoders.bean(MyClass.class);
Dataset results = raw_df.as(myClassEncoder);
List lst = results.collectAsList();
{code}
I do several calculations to get the result I want and the result is correct
all through the way before I collect it.
This is the result for -
{code:java}
results.select(results.col("data").getField("2017-07-01").getField("startTime")).show(false);
{code}
|data[2017-07-01].startTime|data[2017-07-01].endTime|
+-+--+
|1498854000|1498870800 |
This is the result after collecting the reuslts for -
{code:java}
MyClass userData = results.collectAsList().get(0);
MyDTO userDTO = userData.getData().get("2017-07-01");
System.out.println("userDTO startTime: " + userDTO.getSleepStartTime());
System.out.println("userDTO endTime: " + userDTO.getSleepEndTime());
{code}
--
data startTime: 1498870800
data endTime: 1498854000
I tend to believe it is a spark issue. Would love any suggestions on how to
bypass it.
was:
I have the following schema in a dataset -
root
|-- userId: string (nullable = true)
|-- data: map (nullable = true)
||-- key: string
||-- value: struct (valueContainsNull = true)
|||-- startTime: long (nullable = true)
|||-- endTime: long (nullable = true)
|-- offset: long (nullable = true)
And I have the following classes (+ setter and getters which I omitted for
simplicity) -
public class MyClass {
private String userId;
private Map data;
private Long offset;
}
public class MyDTO {
private long startTime;
private long endTime;
}
I collect the result the following way -
Encoder myClassEncoder = Encoders.bean(MyClass.class);
Dataset results = raw_df.as(myClassEncoder);
List lst = results.collectAsList();
I do several calculations to get the result I want and the result is correct
all through the way before I collect it.
This is the result for -
results.select(results.col("data").getField("2017-07-01").getField("startTime")).show(false);
|data[2017-07-01].startTime|data[2017-07-01].endTime|
++--+
|1498854000|1498870800 |
This is the result after collecting the reuslts for -
MyClass userData = results.collectAsList().get(0);
MyDTO userDTO = userData.getData().get("2017-07-01");
System.out.println("userDTO startTime: " + userDTO.getSleepStartTime());
System.out.println("userDTO endTime: " + userDTO.getSleepEndTime());
--
data startTime: 1498870800
data endTime: 1498854000
I tend to believe it is a spark issue. Would love any suggestions on how to
bypass it.
> Java encoders - switch fields on collectAsList
> --
>
> Key: SPARK-21402
> URL: https://issues.apache.org/jira/browse/SPARK-21402
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.1
> Environment: mac os
> spark 2.1.1
> Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_121
>Reporter: Tom
>Priority: Minor
>
> I have the following schema in a dataset -
> root
> |-- userId: string (nullable = true)
> |-- data: map (nullable = true)
> ||-- key: string
> ||-- value: struct (valueContainsNull = true)
> |||-- startTime: long (nullable = true)
> |||-- endTime: long (nullable = true)
> |-- offset: long (nullable = true)
> And I have the following classes (+ setter and getters which I omitted for
> simplicity) -
>
> {code:java}
> public class MyClass {
> private String userId;
> private Map data;
> private Long offset;
> }
> public class MyDTO {
> private long startTime;
> private long endTime;
> }
> {code}
> I collect the result the following way -
> {code:java}
> Encoder myClassEncoder = Encoders.bean(MyClass.class);
> Dataset results = raw_df.as(myClassEncoder);
> List lst = results.collectAsLi
[jira] [Created] (SPARK-21402) Java encoders - switch fields on collectAsList
Tom created SPARK-21402:
---
Summary: Java encoders - switch fields on collectAsList
Key: SPARK-21402
URL: https://issues.apache.org/jira/browse/SPARK-21402
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.1.1
Environment: mac os
spark 2.1.1
Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_121
Reporter: Tom
Priority: Minor
I have the following schema in a dataset -
root
|-- userId: string (nullable = true)
|-- data: map (nullable = true)
||-- key: string
||-- value: struct (valueContainsNull = true)
|||-- startTime: long (nullable = true)
|||-- endTime: long (nullable = true)
|-- offset: long (nullable = true)
And I have the following classes (+ setter and getters which I omitted for
simplicity) -
public class MyClass {
private String userId;
private Map data;
private Long offset;
}
public class MyDTO {
private long startTime;
private long endTime;
}
I collect the result the following way -
Encoder myClassEncoder = Encoders.bean(MyClass.class);
Dataset results = raw_df.as(myClassEncoder);
List lst = results.collectAsList();
I do several calculations to get the result I want and the result is correct
all through the way before I collect it.
This is the result for -
results.select(results.col("data").getField("2017-07-01").getField("startTime")).show(false);
|data[2017-07-01].startTime|data[2017-07-01].endTime|
++--+
|1498854000|1498870800 |
This is the result after collecting the reuslts for -
MyClass userData = results.collectAsList().get(0);
MyDTO userDTO = userData.getData().get("2017-07-01");
System.out.println("userDTO startTime: " + userDTO.getSleepStartTime());
System.out.println("userDTO endTime: " + userDTO.getSleepEndTime());
--
data startTime: 1498870800
data endTime: 1498854000
I tend to believe it is a spark issue. Would love any suggestions on how to
bypass it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
