Can you do a check prior to launching the encryption test to see if the cluster has encryption enabled?
If not, skip the test? On Dec 1, 2014, at 3:12 PM, Andrew Purtell <[email protected]> wrote: >> My cluster won't start now after this IT test runs. That is to be expected? > > Yes, but it is ugly I agree. We can check the deploy-ability of encryption > like we do with compression. Let me open an issue for doing that. > > >> On Dec 1, 2014, at 2:29 AM, Stack <[email protected]> wrote: >> >>> On Sun, Nov 30, 2014 at 7:54 PM, Andrew Purtell <[email protected]> wrote: >>> >>> Yes, you have to set up cluster configuration for encryption or the IT >>> won't work. >> >> >> Ok. Thanks. I'll add note to the IT section that wildcard will run at least >> this test that requires encryption setup. >> >> My cluster won't start now after this IT test runs. That is to be expected? >> >> >> >> >>> Encryption requires creating a keystore, etc. If you run the >>> test as all-localhost, using mvn verify, it should pass, because all >>> configuration can be programatically set up and deployed since the >>> environment is a mini cluster. You can see how the test sets up the >>> necessary configuration using HTU for that case. >> Thanks. Will add note on above and pointers to the security section of the >> refguide (as per your note after this one). >> >> St.Ack >> >> >> >>>> On Sun, Nov 30, 2014 at 6:52 PM, Stack <[email protected]> wrote: >>>> >>>> I just tried running all tests and notice that at least >>>> IntegrationTestIngestWithEcnryption fails reliably with the below. Is >>> this >>>> expected? Does this test require a particular context setup first to run, >>>> one that is not present normally? Once it has run, I cannot get past it. >>>> If I restart servers, they fail on log replay with same exception. Is >>> this >>>> expected? >>>> >>>> On IntegrationTests in general, I was thinking they should generally pass >>>> and that it is a bug if they do not. Is that what others think? >>>> >>>> Has anyone been running IT tests regularly? If so, what has been your >>>> experience? Or if you have been running it tests, do you run individual >>>> tests? >>>> >>>> Thanks. Just trying to figure what current understanding of their state >>> is >>>> before digging in. >>>> Yours, >>>> St.Ack >>>> >>>> 2014-11-29 21:19:06,447 DEBUG >>>> [B.defaultRpcServer.handler=14,queue=2,port=16020] regionserver.HRegion: >>>> Flush requested on >>> IntegrationTestIngestWithEncryption,cccccccc,1417324570806.325074071473eb28b662f7d694e8c609. >>>> 2014-11-29 21:19:06,640 FATAL [MemStoreFlusher.1] >>>> regionserver.HRegionServer: ABORTING region server >>>> c2021.halxg.cloudera.com,16020,1417323325816: >>>> Replay of HLog required. Forcing server shutdown >>>> org.apache.hadoop.hbase.DroppedSnapshotException: region: >>> IntegrationTestIngestWithEncryption,,1417324570806.55e317f49ddf8ed35323d9b675611548. >>>> at >>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1964) >>>> at >>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1730) >>>> at >>> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1662) >>>> at >>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:434) >>>> at >>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:407) >>>> at >>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:69) >>>> at >>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:225) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: >>>> KeyProvider scheme should specify KeyStore type >>>> at >>> org.apache.hadoop.hbase.io.crypto.Encryption.getKeyProvider(Encryption.java:535) >>>> at >>> org.apache.hadoop.hbase.io.crypto.Encryption.getSecretKeyForSubject(Encryption.java:425) >>>> at >>> org.apache.hadoop.hbase.io.crypto.Encryption.encryptWithSubjectKey(Encryption.java:449) >>>> at >>> org.apache.hadoop.hbase.security.EncryptionUtil.wrapKey(EncryptionUtil.java:92) >>>> at >>> org.apache.hadoop.hbase.io.hfile.HFileWriterV3.finishClose(HFileWriterV3.java:127) >>>> at >>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:366) >>>> at >>> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:986) >>>> at >>> org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:67) >>>> at >>> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:80) >>>> at >>>> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:871) >>>> at >>> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2080) >>>> at >>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1918) >>>> ... 7 more >>>> Caused by: java.lang.RuntimeException: KeyProvider scheme should specify >>>> KeyStore type >>>> at >>> org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider.init(KeyStoreKeyProvider.java:142) >>>> at >>> org.apache.hadoop.hbase.io.crypto.Encryption.getKeyProvider(Encryption.java:528) >>>> ... 18 more >>> >>> >>> >>> -- >>> Best regards, >>> >>> - Andy >>> >>> Problems worthy of attack prove their worth by hitting back. - Piet Hein >>> (via Tom White) >>> >
