You can view the status of the Cassandra streaming process to do a one-time rebuild of a node using the Cassandra nodetool utility and/or by monitoring Cassandra log files.
Nodetool to check status
Using nodetool netstats to view streaming activity:
[arris@nxt-failover-eastus2-cass03 conf]# nodetool netstats
Mode: NORMAL
Rebuild bbab1af0-43d8-11e8-9857-cd7a6a718d9e
/10.30.1.31
Sending 149 files, 35336281495 bytes total. Already sent 2 files, 784920988 bytes total
/var/lib/cassandra/data/acdev/fnrfstats-f2ffcd10329011e8a565d1fc89a32cf2/acdev-fnrfstats-ka-446-Data.db 780323369/2009666444 bytes(38%) sent to idx:0/10.30.1.31
/var/lib/cassandra/data/acdev/channeldistdata-feb563e0329011e8a033e712f8dbebb7/acdev-channeldistdata-ka-179-Data.db 1997416/1997416 bytes(100%) sent to idx:0/10.30.1.31
/var/lib/cassandra/data/acdev/channeldistdata-feb563e0329011e8a033e712f8dbebb7/acdev-channeldistdata-ka-133-Data.db 2600203/2600203 bytes(100%) sent to idx:0/10.30.1.31
. . .
[arris@nxt-failover-eastus2-cass03 conf]# nodetool netstats
Mode: NORMAL
Rebuild bbab1af0-43d8-11e8-9857-cd7a6a718d9e
/10.30.1.31
Sending 149 files, 35336281495 bytes total. Already sent 9 files, 13413705415 bytes total
/var/lib/cassandra/data/acdev/fnrfstats-f2ffcd10329011e8a565d1fc89a32cf2/acdev-fnrfstats-ka-388-Data.db 2011812009/2011812009 bytes(100%) sent to idx:0/10.30.1.31
/var/lib/cassandra/data/acdev/fnrfstats-f2ffcd10329011e8a565d1fc89a32cf2/acdev-fnrfstats-ka-301-Data.db 1476219339/2048862132 bytes(72%) sent to idx:0/10.30.1.31
. . .
Log files to check status
Cassandra log files can be found at /var/log/cassandra/system.log.
You can search the logs for the sample messages below to check the rebuild status. This sample displays a log from a new data center node running a rebuild.
[arris@nxt-failover-westus-cass01 logs]# cat /var/log/cassandra/system.log | grep Stream
INFO [RMI TCP Connection(14)-10.30.1.31] 2018-04-19 13:51:17,770 StreamResultFuture.java:86 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e]
Executing streaming plan for Rebuild
INFO [StreamConnectionEstablisher:1] 2018-04-19 13:51:17,771 StreamSession.java:220 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] Starting streaming to /10.20.1.32
INFO [StreamConnectionEstablisher:2] 2018-04-19 13:51:17,772 StreamSession.java:220 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] Starting streaming to /10.20.1.33
INFO [StreamConnectionEstablisher:3] 2018-04-19 13:51:17,772 StreamSession.java:220 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] Starting streaming to /10.20.1.31
INFO [StreamConnectionEstablisher:5] 2018-04-19 13:51:17,772 StreamSession.java:220 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] Starting streaming to /10.30.1.33
INFO [StreamConnectionEstablisher:4] 2018-04-19 13:51:17,772 StreamSession.java:220 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] Starting streaming to /10.30.1.32
INFO [StreamConnectionEstablisher:5] 2018-04-19 13:51:17,788 StreamCoordinator.java:209 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e, ID#0] Beginning stream session with /10.30.1.33
INFO [StreamConnectionEstablisher:4] 2018-04-19 13:51:17,788 StreamCoordinator.java:209 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e, ID#0] Beginning stream session with /10.30.1.32
INFO [StreamConnectionEstablisher:1] 2018-04-19 13:51:17,905 StreamCoordinator.java:209 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e, ID#0] Beginning stream session with /10.20.1.32
INFO [StreamConnectionEstablisher:2] 2018-04-19 13:51:17,905 StreamCoordinator.java:209 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e, ID#0] Beginning stream session with /10.20.1.33
INFO [StreamConnectionEstablisher:3] 2018-04-19 13:51:17,906 StreamCoordinator.java:209 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e, ID#0] Beginning stream session with /10.20.1.31
INFO [STREAM-IN-/10.20.1.32] 2018-04-19 13:51:18,107 StreamResultFuture.java:180 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] Session with /10.20.1.32 is complete
INFO [STREAM-IN-/10.20.1.31] 2018-04-19 13:51:18,110 StreamResultFuture.java:180 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] Session with /10.20.1.31 is complete
INFO [STREAM-IN-/10.30.1.33] 2018-04-19 13:51:18,260 StreamResultFuture.java:166 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e ID#0] Prepare completed. Receiving 7 files(4497627 bytes), sending 0 files(0 bytes)
INFO [STREAM-IN-/10.30.1.32] 2018-04-19 13:51:18,281 StreamResultFuture.java:166 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e ID#0] Prepare completed. Receiving 7 files(4791789 bytes), sending 0 files(0 bytes)
INFO [StreamReceiveTask:2] 2018-04-19 13:51:18,855 StreamResultFuture.java:180 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] Session with /10.30.1.33 is complete
INFO [StreamReceiveTask:5] 2018-04-19 13:51:18,933 StreamResultFuture.java:180 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] Session with /10.30.1.32 is complete
INFO [STREAM-IN-/10.20.1.33] 2018-04-19 13:51:35,013 StreamResultFuture.java:166 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e ID#0] Prepare completed. Receiving 149 files(35336281495 bytes), sending 0 files(0 bytes)
INFO [StreamReceiveTask:17] 2018-04-19 14:19:03,292 StreamResultFuture.java:180 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] Session with /10.20.1.33 is complete
INFO [StreamReceiveTask:17] 2018-04-19 14:19:03,294 StreamResultFuture.java:212 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] All sessions completedFrom one of the nodes in the existing datacenter:
[arris@nxt-failover-eastus2-cass03 conf]# cat /var/log/cassandra/system.log | grep Stream
INFO [main] 2018-03-28 14:02:27,785 StreamResultFuture.java:86 - [Stream #a60476f0-3290-11e8-a033-e712f8dbebb7] Executing streaming plan for Bootstrap
INFO [StreamConnectionEstablisher:1] 2018-03-28 14:02:27,786 StreamSession.java:220 - [Stream #a60476f0-3290-11e8-a033-e712f8dbebb7] Starting streaming to /10.20.1.32
INFO [StreamConnectionEstablisher:2] 2018-03-28 14:02:27,786 StreamSession.java:220 - [Stream #a60476f0-3290-11e8-a033-e712f8dbebb7] Starting streaming to /10.20.1.31
INFO [StreamConnectionEstablisher:1] 2018-03-28 14:02:27,803 StreamCoordinator.java:209 - [Stream #a60476f0-3290-11e8-a033-e712f8dbebb7, ID#0] Beginning stream session with /10.20.1.32
INFO [StreamConnectionEstablisher:2] 2018-03-28 14:02:27,803 StreamCoordinator.java:209 - [Stream #a60476f0-3290-11e8-a033-e712f8dbebb7, ID#0] Beginning stream session with /10.20.1.31
INFO [STREAM-IN-/10.20.1.31] 2018-03-28 14:02:27,829 StreamResultFuture.java:180 - [Stream #a60476f0-3290-11e8-a033-e712f8dbebb7] Session with /10.20.1.31 is complete
INFO [STREAM-IN-/10.20.1.32] 2018-03-28 14:02:27,829 StreamResultFuture.java:180 - [Stream #a60476f0-3290-11e8-a033-e712f8dbebb7] Session with /10.20.1.32 is complete
INFO [STREAM-IN-/10.20.1.32] 2018-03-28 14:02:27,840 StreamResultFuture.java:212 - [Stream #a60476f0-3290-11e8-a033-e712f8dbebb7] All sessions completed
INFO [STREAM-IN-/10.20.1.31] 2018-03-28 14:02:27,851 StreamResultFuture.java:212 - [Stream #a60476f0-3290-11e8-a033-e712f8dbebb7] All sessions completed
INFO [STREAM-INIT-/10.30.1.31:35058] 2018-04-19 13:51:17,880 StreamResultFuture.java:109 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e ID#0] Creating new streaming plan for Rebuild
INFO [STREAM-INIT-/10.30.1.31:35058] 2018-04-19 13:51:17,892 StreamResultFuture.java:116 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e, ID#0] Received streaming plan for Rebuild
INFO [STREAM-INIT-/10.30.1.31:50289] 2018-04-19 13:51:17,942 StreamResultFuture.java:116 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e, ID#0] Received streaming plan for Rebuild
INFO [STREAM-IN-/10.30.1.31] 2018-04-19 13:51:34,975 StreamResultFuture.java:166 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 149 files(35336281495 bytes)
INFO [STREAM-IN-/10.30.1.31] 2018-04-19 14:19:03,329 StreamResultFuture.java:180 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] Session with /10.30.1.31 is complete
INFO [STREAM-IN-/10.30.1.31] 2018-04-19 14:19:03,331 StreamResultFuture.java:212 - [Stream #bbab1af0-43d8-11e8-9857-cd7a6a718d9e] All sessions completed
