[jira] [Commented] (CASSANDRA-8356) Slice query on a super column family with counters doesn't get all the data
[
https://issues.apache.org/jira/browse/CASSANDRA-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221557#comment-14221557
]
Aleksey Yeschenko commented on CASSANDRA-8356:
--
Looks like another (hopefully the last one this time) gift from CASSANDRA-3237.
Can you try to reproduce with regular, non-counter columns, but otherwise same
schema/data? I don't think this is going to be counters specific.
Thanks.
> Slice query on a super column family with counters doesn't get all the data
> ---
>
> Key: CASSANDRA-8356
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8356
> Project: Cassandra
> Issue Type: Bug
>Reporter: Nicolas Lalevée
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.12
>
>
> We've finally been able to upgrade our cluster to 2.0.11, after
> CASSANDRA-7188 being fixed.
> But now slice queries on a super column family with counters doesn't return
> all the expected data. We first though because of all the trouble we had that
> we lost data, but there a way to actually get the data, so nothing is lost;
> it just that cassandra seems to incorrectly skip it.
> See the following CQL log:
> {noformat}
> cqlsh:Theme> desc table theme_view;
> CREATE TABLE theme_view (
> key bigint,
> column1 varint,
> column2 text,
> value counter,
> PRIMARY KEY ((key), column1, column2)
> ) WITH COMPACT STORAGE AND
> bloom_filter_fp_chance=0.01 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> index_interval=128 AND
> read_repair_chance=1.00 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> default_time_to_live=0 AND
> speculative_retry='99.0PERCENTILE' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'SnappyCompressor'};
> cqlsh:Theme> select * from theme_view where key = 99421 limit 10;
> key | column1 | column2| value
> ---+-++---
> 99421 | -12 | 2011-03-25 |59
> 99421 | -12 | 2011-03-26 | 5
> 99421 | -12 | 2011-03-27 | 2
> 99421 | -12 | 2011-03-28 |40
> 99421 | -12 | 2011-03-29 |14
> 99421 | -12 | 2011-03-30 |17
> 99421 | -12 | 2011-03-31 | 5
> 99421 | -12 | 2011-04-01 |37
> 99421 | -12 | 2011-04-02 | 7
> 99421 | -12 | 2011-04-03 | 4
> (10 rows)
> cqlsh:Theme> select * from theme_view where key = 99421 and column1 = -12
> limit 10;
> key | column1 | column2| value
> ---+-++---
> 99421 | -12 | 2011-03-25 |59
> 99421 | -12 | 2014-05-06 |15
> 99421 | -12 | 2014-06-06 | 7
> 99421 | -12 | 2014-06-10 |22
> 99421 | -12 | 2014-06-11 |34
> 99421 | -12 | 2014-06-12 |35
> 99421 | -12 | 2014-06-13 |26
> 99421 | -12 | 2014-06-14 |16
> 99421 | -12 | 2014-06-15 |24
> 99421 | -12 | 2014-06-16 |25
> (10 rows)
> {noformat}
> As you can see the second query should return data from 2012, but it is not.
> Via thrift, we have the exact same bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8356) Slice query on a super column family with counters doesn't get all the data
[
https://issues.apache.org/jira/browse/CASSANDRA-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14222950#comment-14222950
]
Nicolas Lalevée commented on CASSANDRA-8356:
I got the snapshot data from a node on my local machine, and I tried to load it
up in a local cassandra node 2.0.11.
The node did the "opening" of the files correctly. But querying against it is
impossible, I hit the following error:
{noformat}
ERROR 11:28:45,693 Exception in thread Thread[ReadStage:2,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1981)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:267)
at
org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:587)
at
org.apache.cassandra.utils.ByteBufferUtil.readBytesWithShortLength(ByteBufferUtil.java:596)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:61)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:1)
at
org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:436)
at
org.apache.cassandra.db.RangeTombstoneList.add(RangeTombstoneList.java:141)
at
org.apache.cassandra.db.RangeTombstoneList.add(RangeTombstoneList.java:113)
at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:202)
at
org.apache.cassandra.db.AbstractThreadUnsafeSortedColumns.delete(AbstractThreadUnsafeSortedColumns.java:54)
at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:155)
at
org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:168)
at
org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.(MergeIterator.java:87)
at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:120)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:56)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1413)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1977)
... 3 more
{nofomart}
This reminded me of an error we had on our test cluster, when we tested the
upgrade to 2.0.x : CASSANDRA-6733
So here, I ran an upgradesstable on our production cluster, and now the slice
queries return all the expected data. So everything is back to normal (and I am
very pleased by the lower cpu activity with 2.0.x for the same load).
I looked up again the logs in prod, I still don't see any such Buffer.limit
errors. I don't know what was going wrong.
As for CASSANDRA-6733, I have a snapshot of the data before the upgrade_sstable
(unfortunately I don't have a snapshot pre-upgrade, but somme sstables are sill
in the old format). If someone wants the data to analyse it, concat me,
nlalevee at scoop.it.
> Slice query on a super column family with counters doesn't get all the data
> ---
>
> Key: CASSANDRA-8356
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8356
> Project: Cassandra
> Issue Type: Bug
>Reporter: Nicolas Lalevée
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.12
>
>
> We've finally been able to upgrade our cluster to 2.0.11, after
> CASSANDRA-7188 being fixed.
> But now slice queries on a super column family with counters doesn't return
> all the expected data. We first though because of all the trouble we had that
> we lost data, but there a
[jira] [Commented] (CASSANDRA-8356) Slice query on a super column family with counters doesn't get all the data
[
https://issues.apache.org/jira/browse/CASSANDRA-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238404#comment-14238404
]
Aleksey Yeschenko commented on CASSANDRA-8356:
--
[~philipthompson] Philip, could you please try to repro? With counters and with
regular columns.
Create a supercolumn table on 1.2, write a bunch of data in the same pattern as
in the ticket, flush them all (and save somewhere). Upgrade the node to 2.0
without upgrading the sstables, and try the same queries - see if everything is
still there.
Thanks.
> Slice query on a super column family with counters doesn't get all the data
> ---
>
> Key: CASSANDRA-8356
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8356
> Project: Cassandra
> Issue Type: Bug
>Reporter: Nicolas Lalevée
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.12
>
>
> We've finally been able to upgrade our cluster to 2.0.11, after
> CASSANDRA-7188 being fixed.
> But now slice queries on a super column family with counters doesn't return
> all the expected data. We first though because of all the trouble we had that
> we lost data, but there a way to actually get the data, so nothing is lost;
> it just that cassandra seems to incorrectly skip it.
> See the following CQL log:
> {noformat}
> cqlsh:Theme> desc table theme_view;
> CREATE TABLE theme_view (
> key bigint,
> column1 varint,
> column2 text,
> value counter,
> PRIMARY KEY ((key), column1, column2)
> ) WITH COMPACT STORAGE AND
> bloom_filter_fp_chance=0.01 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> index_interval=128 AND
> read_repair_chance=1.00 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> default_time_to_live=0 AND
> speculative_retry='99.0PERCENTILE' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'SnappyCompressor'};
> cqlsh:Theme> select * from theme_view where key = 99421 limit 10;
> key | column1 | column2| value
> ---+-++---
> 99421 | -12 | 2011-03-25 |59
> 99421 | -12 | 2011-03-26 | 5
> 99421 | -12 | 2011-03-27 | 2
> 99421 | -12 | 2011-03-28 |40
> 99421 | -12 | 2011-03-29 |14
> 99421 | -12 | 2011-03-30 |17
> 99421 | -12 | 2011-03-31 | 5
> 99421 | -12 | 2011-04-01 |37
> 99421 | -12 | 2011-04-02 | 7
> 99421 | -12 | 2011-04-03 | 4
> (10 rows)
> cqlsh:Theme> select * from theme_view where key = 99421 and column1 = -12
> limit 10;
> key | column1 | column2| value
> ---+-++---
> 99421 | -12 | 2011-03-25 |59
> 99421 | -12 | 2014-05-06 |15
> 99421 | -12 | 2014-06-06 | 7
> 99421 | -12 | 2014-06-10 |22
> 99421 | -12 | 2014-06-11 |34
> 99421 | -12 | 2014-06-12 |35
> 99421 | -12 | 2014-06-13 |26
> 99421 | -12 | 2014-06-14 |16
> 99421 | -12 | 2014-06-15 |24
> 99421 | -12 | 2014-06-16 |25
> (10 rows)
> {noformat}
> As you can see the second query should return data from 2012, but it is not.
> Via thrift, we have the exact same bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8356) Slice query on a super column family with counters doesn't get all the data
[
https://issues.apache.org/jira/browse/CASSANDRA-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14241116#comment-14241116
]
Philo Yang commented on CASSANDRA-8356:
---
Hi, in fact I have the same trouble for missing data on slice query. My cluster
is 2.1.1, and regular table(create by cql3, no counter). My table is like this:
{noformat}
CREATE TABLE word (
user text,
word text,
alter_time bigint,
(some other column...)
PRIMARY KEY (user, word)
) WITH CLUSTERING ORDER BY (word ASC)
{noformat}
If a row with 26 column whose column names are "a","b"..."z".
Usually I query this table like this:
{noformat}
select * from word where user = 'userid';
{noformat}
and it should return 26 rows in cql.
However, in some rows (most of rows in this table won't lose data) it return
only part of columns, some row(row for cql, column for cassandra ), will not
return even in consistency level ALL. The row that doesn't return is fixed for
querying many times. For example, "b", is a row that is missing. If I query like
{noformat}
select * from word where user = 'userid';
or
select * from word where user = 'userid' and word >'a';
or
select * from word where user = 'userid' and word <='b';
or
select * from word where user = 'userid' and word <='b' order by word desc;
or
select * from word where user = 'userid' and word <'z';
{noformat}
'b' is always missing.
But if I query like:
{noformat}
select * from word where user = 'userid' and word ='b';
or
select * from word where user = 'userid' and word >='b';
or
select * from word where user = 'userid' and word >='b' order by word desc;
{noformat}
It will show in result set.
> Slice query on a super column family with counters doesn't get all the data
> ---
>
> Key: CASSANDRA-8356
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8356
> Project: Cassandra
> Issue Type: Bug
>Reporter: Nicolas Lalevée
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.12
>
>
> We've finally been able to upgrade our cluster to 2.0.11, after
> CASSANDRA-7188 being fixed.
> But now slice queries on a super column family with counters doesn't return
> all the expected data. We first though because of all the trouble we had that
> we lost data, but there a way to actually get the data, so nothing is lost;
> it just that cassandra seems to incorrectly skip it.
> See the following CQL log:
> {noformat}
> cqlsh:Theme> desc table theme_view;
> CREATE TABLE theme_view (
> key bigint,
> column1 varint,
> column2 text,
> value counter,
> PRIMARY KEY ((key), column1, column2)
> ) WITH COMPACT STORAGE AND
> bloom_filter_fp_chance=0.01 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> index_interval=128 AND
> read_repair_chance=1.00 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> default_time_to_live=0 AND
> speculative_retry='99.0PERCENTILE' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'SnappyCompressor'};
> cqlsh:Theme> select * from theme_view where key = 99421 limit 10;
> key | column1 | column2| value
> ---+-++---
> 99421 | -12 | 2011-03-25 |59
> 99421 | -12 | 2011-03-26 | 5
> 99421 | -12 | 2011-03-27 | 2
> 99421 | -12 | 2011-03-28 |40
> 99421 | -12 | 2011-03-29 |14
> 99421 | -12 | 2011-03-30 |17
> 99421 | -12 | 2011-03-31 | 5
> 99421 | -12 | 2011-04-01 |37
> 99421 | -12 | 2011-04-02 | 7
> 99421 | -12 | 2011-04-03 | 4
> (10 rows)
> cqlsh:Theme> select * from theme_view where key = 99421 and column1 = -12
> limit 10;
> key | column1 | column2| value
> ---+-++---
> 99421 | -12 | 2011-03-25 |59
> 99421 | -12 | 2014-05-06 |15
> 99421 | -12 | 2014-06-06 | 7
> 99421 | -12 | 2014-06-10 |22
> 99421 | -12 | 2014-06-11 |34
> 99421 | -12 | 2014-06-12 |35
> 99421 | -12 | 2014-06-13 |26
> 99421 | -12 | 2014-06-14 |16
> 99421 | -12 | 2014-06-15 |24
> 99421 | -12 | 2014-06-16 |25
> (10 rows)
> {noformat}
> As you can see the second query should return data from 2012, but it is not.
> Via thrift, we have the exact same bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8356) Slice query on a super column family with counters doesn't get all the data
[
https://issues.apache.org/jira/browse/CASSANDRA-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14246971#comment-14246971
]
Philip Thompson commented on CASSANDRA-8356:
[~iamaleksey], I sent you an email, but what I have so far is here:
https://github.com/riptano/cassandra-dtest/blob/8356/upgrade_supercolumns_test.py
I am not reproducing the problem, but the schemas match, so I only assume I
am inserting data in an incorrect pattern. Do you see anything obviously
incorrect with the tests?
> Slice query on a super column family with counters doesn't get all the data
> ---
>
> Key: CASSANDRA-8356
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8356
> Project: Cassandra
> Issue Type: Bug
>Reporter: Nicolas Lalevée
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.12
>
>
> We've finally been able to upgrade our cluster to 2.0.11, after
> CASSANDRA-7188 being fixed.
> But now slice queries on a super column family with counters doesn't return
> all the expected data. We first though because of all the trouble we had that
> we lost data, but there a way to actually get the data, so nothing is lost;
> it just that cassandra seems to incorrectly skip it.
> See the following CQL log:
> {noformat}
> cqlsh:Theme> desc table theme_view;
> CREATE TABLE theme_view (
> key bigint,
> column1 varint,
> column2 text,
> value counter,
> PRIMARY KEY ((key), column1, column2)
> ) WITH COMPACT STORAGE AND
> bloom_filter_fp_chance=0.01 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> index_interval=128 AND
> read_repair_chance=1.00 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> default_time_to_live=0 AND
> speculative_retry='99.0PERCENTILE' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'SnappyCompressor'};
> cqlsh:Theme> select * from theme_view where key = 99421 limit 10;
> key | column1 | column2| value
> ---+-++---
> 99421 | -12 | 2011-03-25 |59
> 99421 | -12 | 2011-03-26 | 5
> 99421 | -12 | 2011-03-27 | 2
> 99421 | -12 | 2011-03-28 |40
> 99421 | -12 | 2011-03-29 |14
> 99421 | -12 | 2011-03-30 |17
> 99421 | -12 | 2011-03-31 | 5
> 99421 | -12 | 2011-04-01 |37
> 99421 | -12 | 2011-04-02 | 7
> 99421 | -12 | 2011-04-03 | 4
> (10 rows)
> cqlsh:Theme> select * from theme_view where key = 99421 and column1 = -12
> limit 10;
> key | column1 | column2| value
> ---+-++---
> 99421 | -12 | 2011-03-25 |59
> 99421 | -12 | 2014-05-06 |15
> 99421 | -12 | 2014-06-06 | 7
> 99421 | -12 | 2014-06-10 |22
> 99421 | -12 | 2014-06-11 |34
> 99421 | -12 | 2014-06-12 |35
> 99421 | -12 | 2014-06-13 |26
> 99421 | -12 | 2014-06-14 |16
> 99421 | -12 | 2014-06-15 |24
> 99421 | -12 | 2014-06-16 |25
> (10 rows)
> {noformat}
> As you can see the second query should return data from 2012, but it is not.
> Via thrift, we have the exact same bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8356) Slice query on a super column family with counters doesn't get all the data
[
https://issues.apache.org/jira/browse/CASSANDRA-8356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247062#comment-14247062
]
Philip Thompson commented on CASSANDRA-8356:
I have also tried inserting data after the upgrade, so the slice needs to read
from 2.0 and 1.2 sstables, but that had no effect. I still cannot reproduce.
> Slice query on a super column family with counters doesn't get all the data
> ---
>
> Key: CASSANDRA-8356
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8356
> Project: Cassandra
> Issue Type: Bug
>Reporter: Nicolas Lalevée
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.12
>
>
> We've finally been able to upgrade our cluster to 2.0.11, after
> CASSANDRA-7188 being fixed.
> But now slice queries on a super column family with counters doesn't return
> all the expected data. We first though because of all the trouble we had that
> we lost data, but there a way to actually get the data, so nothing is lost;
> it just that cassandra seems to incorrectly skip it.
> See the following CQL log:
> {noformat}
> cqlsh:Theme> desc table theme_view;
> CREATE TABLE theme_view (
> key bigint,
> column1 varint,
> column2 text,
> value counter,
> PRIMARY KEY ((key), column1, column2)
> ) WITH COMPACT STORAGE AND
> bloom_filter_fp_chance=0.01 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> index_interval=128 AND
> read_repair_chance=1.00 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> default_time_to_live=0 AND
> speculative_retry='99.0PERCENTILE' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'SnappyCompressor'};
> cqlsh:Theme> select * from theme_view where key = 99421 limit 10;
> key | column1 | column2| value
> ---+-++---
> 99421 | -12 | 2011-03-25 |59
> 99421 | -12 | 2011-03-26 | 5
> 99421 | -12 | 2011-03-27 | 2
> 99421 | -12 | 2011-03-28 |40
> 99421 | -12 | 2011-03-29 |14
> 99421 | -12 | 2011-03-30 |17
> 99421 | -12 | 2011-03-31 | 5
> 99421 | -12 | 2011-04-01 |37
> 99421 | -12 | 2011-04-02 | 7
> 99421 | -12 | 2011-04-03 | 4
> (10 rows)
> cqlsh:Theme> select * from theme_view where key = 99421 and column1 = -12
> limit 10;
> key | column1 | column2| value
> ---+-++---
> 99421 | -12 | 2011-03-25 |59
> 99421 | -12 | 2014-05-06 |15
> 99421 | -12 | 2014-06-06 | 7
> 99421 | -12 | 2014-06-10 |22
> 99421 | -12 | 2014-06-11 |34
> 99421 | -12 | 2014-06-12 |35
> 99421 | -12 | 2014-06-13 |26
> 99421 | -12 | 2014-06-14 |16
> 99421 | -12 | 2014-06-15 |24
> 99421 | -12 | 2014-06-16 |25
> (10 rows)
> {noformat}
> As you can see the second query should return data from 2012, but it is not.
> Via thrift, we have the exact same bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
