[GitHub] nifi issue #1376: NIFI-3268 Add AUTO_INCREMENT column in GenerateTableFetch ...
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1376 @qfdk Sometimes, I encounter issue while debugging, such as forgetting terminate old IntelliJ debugger or having other thread being stopped at different break point, then NiFi jetty thread can't respond. If NiFi UI keeps showing a little running circle and doesn't update, probably that is the case. Otherwise, I've been able to debug it with IntelliJ fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3268) Add AUTO_INCREMENT column in GenerateTableFetch to benefit index
[ https://issues.apache.org/jira/browse/NIFI-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952974#comment-15952974 ] ASF GitHub Bot commented on NIFI-3268: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1376 @qfdk Sometimes, I encounter issue while debugging, such as forgetting terminate old IntelliJ debugger or having other thread being stopped at different break point, then NiFi jetty thread can't respond. If NiFi UI keeps showing a little running circle and doesn't update, probably that is the case. Otherwise, I've been able to debug it with IntelliJ fine. > Add AUTO_INCREMENT column in GenerateTableFetch to benefit index > > > Key: NIFI-3268 > URL: https://issues.apache.org/jira/browse/NIFI-3268 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.1.1 > Environment: - ubuntu 16.04 > - java version "1.8.0_111" > - Java(TM) SE Runtime Environment (build 1.8.0_111-b14) > - Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode) >Reporter: qfdk > Labels: easyfix > > I added AUTO_INCREMENT column in GenerateTableFetch to benefit index column > By default this processor uses OFFSET, i have problems with large data. > somme column has index so we could use index to speed up query time. > I posted question here : > https://community.hortonworks.com/questions/72586/how-can-i-use-an-array-with-putelasticsearch.html > If you indexed un column (id), you could use this sql > ``` > select xxx > From x > where 20=>id > order by id > limit 20 > ``` > “OFFSET is bad for skipping previous rows.” [Online]. Available: > http://Use-The-Index-Luke.com/sql/partial-results/fetch-next-page. [Accessed: > 27-Dec-2016]. > Thank you in advance -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3586) Nifi is not returning PID in Windows
[ https://issues.apache.org/jira/browse/NIFI-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952859#comment-15952859 ] ASF GitHub Bot commented on NIFI-3586: -- Github user trkurc commented on a diff in the pull request: https://github.com/apache/nifi/pull/1586#discussion_r109320687 --- Diff: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/util/OSUtil.java --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.bootstrap.util; + +import java.lang.reflect.Field; + +import org.slf4j.Logger; +import com.sun.jna.Pointer; +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.WinNT; + +/** + * OS specific utilities with generic method interfaces + */ +public final class OSUtil { + +private static Long getUnicesPid(final Process process, final Logger logger) { +try { +final Class procClass = process.getClass(); +final Field pidField = procClass.getDeclaredField("pid"); +pidField.setAccessible(true); +final Object pidObject = pidField.get(process); + +logger.debug("PID Object = {}", pidObject); + +if (pidObject instanceof Number) { +return ((Number) pidObject).longValue(); +} +return null; +} catch (final IllegalAccessException | NoSuchFieldException nsfe) { +logger.debug("Could not find PID for child process due to {}", nsfe); +return null; +} +} + +private static Long getWindowsProcessId(final Process process, final Logger logger) { +/* determine the pid on windows plattforms */ +try { +Field f = process.getClass().getDeclaredField("handle"); +f.setAccessible(true); +long handl = f.getLong(process); + +Kernel32 kernel = Kernel32.INSTANCE; +WinNT.HANDLE handle = new WinNT.HANDLE(); +handle.setPointer(Pointer.createConstant(handl)); +int ret = kernel.GetProcessId(handle); +logger.debug("Detected pid: {}", ret); +return Long.valueOf(ret); +} catch (final IllegalAccessException | NoSuchFieldException nsfe) { +logger.debug("Could not find PID for child process due to {}", nsfe); +} +return null; +} + +public static Long getProcessId(final Process process, final Logger logger) { --- End diff -- I know it wasn't there before when this was a private method before, but I'd LOVE to see a javadoc comment to describe the contract for this method. I think returning null is a bit counter-intuitive > Nifi is not returning PID in Windows > > > Key: NIFI-3586 > URL: https://issues.apache.org/jira/browse/NIFI-3586 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0, 0.5.0, 0.6.0, 0.7.0, 1.2.0, 1.1.1, 1.0.1 > Environment: Java <=8, Windows >Reporter: Puspendu Banerjee >Priority: Minor > > Nifi PID is unavailable during startup under Windows -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1586: NIFI-3586: Fix for retrieving ProcessID for NiFi un...
Github user trkurc commented on a diff in the pull request: https://github.com/apache/nifi/pull/1586#discussion_r109320687 --- Diff: nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/util/OSUtil.java --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.bootstrap.util; + +import java.lang.reflect.Field; + +import org.slf4j.Logger; +import com.sun.jna.Pointer; +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.WinNT; + +/** + * OS specific utilities with generic method interfaces + */ +public final class OSUtil { + +private static Long getUnicesPid(final Process process, final Logger logger) { +try { +final Class procClass = process.getClass(); +final Field pidField = procClass.getDeclaredField("pid"); +pidField.setAccessible(true); +final Object pidObject = pidField.get(process); + +logger.debug("PID Object = {}", pidObject); + +if (pidObject instanceof Number) { +return ((Number) pidObject).longValue(); +} +return null; +} catch (final IllegalAccessException | NoSuchFieldException nsfe) { +logger.debug("Could not find PID for child process due to {}", nsfe); +return null; +} +} + +private static Long getWindowsProcessId(final Process process, final Logger logger) { +/* determine the pid on windows plattforms */ +try { +Field f = process.getClass().getDeclaredField("handle"); +f.setAccessible(true); +long handl = f.getLong(process); + +Kernel32 kernel = Kernel32.INSTANCE; +WinNT.HANDLE handle = new WinNT.HANDLE(); +handle.setPointer(Pointer.createConstant(handl)); +int ret = kernel.GetProcessId(handle); +logger.debug("Detected pid: {}", ret); +return Long.valueOf(ret); +} catch (final IllegalAccessException | NoSuchFieldException nsfe) { +logger.debug("Could not find PID for child process due to {}", nsfe); +} +return null; +} + +public static Long getProcessId(final Process process, final Logger logger) { --- End diff -- I know it wasn't there before when this was a private method before, but I'd LOVE to see a javadoc comment to describe the contract for this method. I think returning null is a bit counter-intuitive --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3660) ClassCastException from ConvertAvroToORC when the schema contains a map with an array value
[ https://issues.apache.org/jira/browse/NIFI-3660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952855#comment-15952855 ] Steve Champagne commented on NIFI-3660: --- I was doing some research on this and it looks like this particular exception is caused by what I think is a typo on line 152 of NiFiOrcUtils where getMapKeyTypeInfo should be getMapValueTypeInfo instead. When I change that I get an error stating that maps can only contain Writable types. Would there be any issues if arrays were converted to ArrayWritables? > ClassCastException from ConvertAvroToORC when the schema contains a map with > an array value > --- > > Key: NIFI-3660 > URL: https://issues.apache.org/jira/browse/NIFI-3660 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.1.1 >Reporter: Steve Champagne > Attachments: PrimitiveToListTypeException.xml > > > I am getting the following exception with the attached template. > {panel} > java.lang.ClassCastException: > org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to > org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo > at > org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:143) > ~[na:na] > at > org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.lambda$convertToORCObject$8(NiFiOrcUtils.java:157) > ~[na:na] > at java.util.HashMap.forEach(HashMap.java:1288) ~[na:1.8.0_111] > at > org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:155) > ~[na:na] > at > org.apache.nifi.processors.hive.ConvertAvroToORC.lambda$onTrigger$0(ConvertAvroToORC.java:243) > ~[na:na] > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2578) > ~[nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] > at > org.apache.nifi.processors.hive.ConvertAvroToORC.onTrigger(ConvertAvroToORC.java:207) > ~[na:na] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) > ~[nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) > [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_111] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_111] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_111] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_111] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_111] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_111] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > {panel} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3666) Skipped tests on windows need to be validated or fixed
[ https://issues.apache.org/jira/browse/NIFI-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952799#comment-15952799 ] Joseph Percivall commented on NIFI-3666: [~joewitt], I checked out the AttributeRollingWindow tests and I'm confused about why they need to be ignored on Windows. I am able to run them just fine on my system (Win 10). The tests are inherently time related, due to the nature of the processor, so I'd understand if the tests should be ignored/reworked due to that but I don't see any Windows specific reworks that are needed. > Skipped tests on windows need to be validated or fixed > -- > > Key: NIFI-3666 > URL: https://issues.apache.org/jira/browse/NIFI-3666 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions >Reporter: Joseph Witt >Priority: Critical > > In NIFI-3440 a number of relatively recently created tests were failing on > windows. These tests were skipped when running on windows to help keep the > build moving along and to continue to test regressions on older more stable > tests. However, this approach leaves room for error because we must go > through each and validate whether it was a bad test that needs to be fixed to > be more stable/portable OR whether the test exposed a bug in the code and its > behavior on windows. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3644) Add DetectDuplicateUsingHBase processor
[ https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952671#comment-15952671 ] ASF GitHub Bot commented on NIFI-3644: -- GitHub user baolsen opened a pull request: https://github.com/apache/nifi/pull/1645 NIFI-3644 - Added HBase_1_1_2_ClientMapCacheService Added HBase_1_1_2_ClientMapCacheService which implements DistributedMapCacheClient. The DetectDuplicate processor can now make use of HBase_1_1_2_ClientMapCacheService for storing the duplicate cache on HBase. You can merge this pull request into a Git repository by running: $ git pull https://github.com/baolsen/nifi DistributedMapCacheHBaseClientService Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1645.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1645 commit 8c0285b5efb6afd1607bb050650b758fed7d06e3 Author: baolsenDate: 2017-03-23T12:35:43Z Update HBaseClientService.java Added "get" function call for doing single row lookup on HBase (HBase get) commit 03d1b36376c6954d8bdcf4056314fced0cf0d1fc Author: baolsen Date: 2017-03-23T13:20:41Z Update HBase_1_1_2_ClientService.java Implemented "get" function for retrieval of single HBase rows. commit 6dbca10e82b3b6b8ac94f8f0152b8fff85008082 Author: baolsen Date: 2017-03-23T13:33:15Z Update HBase_1_1_2_ClientService.java commit df30a22a3ba71fedfe1dffedefcc0eb64c3670b0 Author: baolsen Date: 2017-03-23T13:40:08Z Update HBase_1_1_2_ClientService.java commit 6d8036cc03ef49e41b92dbb5fa7e0de41cc15c3d Author: baolsen Date: 2017-03-23T13:44:12Z Update MockHBaseClientService.java Implemented "get" function with UnsupportedException commit 4bcb26fd6a99a23852097f4f3db02cbeb6b8a3b5 Author: baolsen Date: 2017-03-23T13:46:23Z Update HBase_1_1_2_ClientService.java commit 4b266d9d1d112e2bf8aa198f87253d17c055dbbc Author: baolsen Date: 2017-03-23T13:50:09Z Update MockHBaseClientService.java commit 2ef850bc7c2bce5f9dd35fc9ce5cf08c7ecf07c4 Author: baolsen Date: 2017-03-29T08:51:11Z Test commit e802f147bcd19664b9053e240ec1476ff7a61e7b Author: baolsen Date: 2017-03-29T08:52:35Z Test commit 4cabff26658090c08d813e74d27894a9fd684c57 Author: baolsen Date: 2017-03-31T07:59:50Z Completed initial development of HBase_1_1_2_ClientMapCacheService.java which is compatible with DetectDuplicate (and other processors) Still need to implement value deletion commit 7790d3f5a8d56f0801d40ad2c836a8db7c123e1b Author: baolsen Date: 2017-03-31T08:31:06Z Undid changes to files for an earlier attempt at this commit 594dc059cdbe708f10849c794b826d24e83e787d Author: baolsen Date: 2017-03-31T08:33:47Z Undid changes to files for an earlier attempt at this commit fbd3034e736ecdd1d721cc788e5c984eee6560c7 Author: baolsen Date: 2017-04-02T13:01:21Z Added remove() for cache and Documentation > Add DetectDuplicateUsingHBase processor > --- > > Key: NIFI-3644 > URL: https://issues.apache.org/jira/browse/NIFI-3644 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bjorn Olsen >Priority: Minor > > The DetectDuplicate processor makes use of a distributed map cache for > maintaining a list of unique file identifiers (such as hashes). > The distributed map cache functionality could be provided by an HBase table, > which then allows for reliably storing a huge volume of file identifiers and > auditing information. The downside of this approach is of course that HBase > is required. > Storing the unique file identifiers in a reliable, query-able manner along > with some audit information is of benefit to several use cases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1645: NIFI-3644 - Added HBase_1_1_2_ClientMapCacheService
GitHub user baolsen opened a pull request: https://github.com/apache/nifi/pull/1645 NIFI-3644 - Added HBase_1_1_2_ClientMapCacheService Added HBase_1_1_2_ClientMapCacheService which implements DistributedMapCacheClient. The DetectDuplicate processor can now make use of HBase_1_1_2_ClientMapCacheService for storing the duplicate cache on HBase. You can merge this pull request into a Git repository by running: $ git pull https://github.com/baolsen/nifi DistributedMapCacheHBaseClientService Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1645.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1645 commit 8c0285b5efb6afd1607bb050650b758fed7d06e3 Author: baolsenDate: 2017-03-23T12:35:43Z Update HBaseClientService.java Added "get" function call for doing single row lookup on HBase (HBase get) commit 03d1b36376c6954d8bdcf4056314fced0cf0d1fc Author: baolsen Date: 2017-03-23T13:20:41Z Update HBase_1_1_2_ClientService.java Implemented "get" function for retrieval of single HBase rows. commit 6dbca10e82b3b6b8ac94f8f0152b8fff85008082 Author: baolsen Date: 2017-03-23T13:33:15Z Update HBase_1_1_2_ClientService.java commit df30a22a3ba71fedfe1dffedefcc0eb64c3670b0 Author: baolsen Date: 2017-03-23T13:40:08Z Update HBase_1_1_2_ClientService.java commit 6d8036cc03ef49e41b92dbb5fa7e0de41cc15c3d Author: baolsen Date: 2017-03-23T13:44:12Z Update MockHBaseClientService.java Implemented "get" function with UnsupportedException commit 4bcb26fd6a99a23852097f4f3db02cbeb6b8a3b5 Author: baolsen Date: 2017-03-23T13:46:23Z Update HBase_1_1_2_ClientService.java commit 4b266d9d1d112e2bf8aa198f87253d17c055dbbc Author: baolsen Date: 2017-03-23T13:50:09Z Update MockHBaseClientService.java commit 2ef850bc7c2bce5f9dd35fc9ce5cf08c7ecf07c4 Author: baolsen Date: 2017-03-29T08:51:11Z Test commit e802f147bcd19664b9053e240ec1476ff7a61e7b Author: baolsen Date: 2017-03-29T08:52:35Z Test commit 4cabff26658090c08d813e74d27894a9fd684c57 Author: baolsen Date: 2017-03-31T07:59:50Z Completed initial development of HBase_1_1_2_ClientMapCacheService.java which is compatible with DetectDuplicate (and other processors) Still need to implement value deletion commit 7790d3f5a8d56f0801d40ad2c836a8db7c123e1b Author: baolsen Date: 2017-03-31T08:31:06Z Undid changes to files for an earlier attempt at this commit 594dc059cdbe708f10849c794b826d24e83e787d Author: baolsen Date: 2017-03-31T08:33:47Z Undid changes to files for an earlier attempt at this commit fbd3034e736ecdd1d721cc788e5c984eee6560c7 Author: baolsen Date: 2017-04-02T13:01:21Z Added remove() for cache and Documentation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---