Hi all, This is test report for ConnMan upstream 0.53.13. In this testing, we ran 331 cases. 306 cases passed and 25 case failed because of known/new bug. The pass rate is 92%. We found 2 new bugs, which were soon fixed.
The testing for more recent package shows state issues mostly got fixed, but we see some new issues, which need to be double check next week. Verified Bugs ============================================= [437] ConnMan needs to connect hidden service automatically http://bugs.meego.com/show_bug.cgi?id=437 New Bugs ============================================= [2846] Switch between two hidden wifi services, sometimes one of them will fail to be connected anymore http://bugs.meego.com/show_bug.cgi?id=2846 [2850] ConnMan easily fail to connect to hidden WIFI services for the first time after reboot http://bugs.meego.com/show_bug.cgi?id=2850 Testing Environment ============================================== Hardware: netbook ACER AspireOne NAV50/Eeepc 1005PE/AspireOne ZG5 Image: MeeGo release 1.0/Build 20100524.1 ConnMan: connman-0.53.13.g1c18278-1.1 3G: Unicom Sim card/Dell 5400 BT: AnyCom AP/CSR dongle (disable the onboard BT device) Wireless AP: Linksys/Asus/Cisco 1252 Test Execution Summary ============================================== >From Carrick UI: Feature Total Pass Fail Comments Wire 2 2 WiFi 55 50 5 #437 #610 #2640 3G 14 14 BT 5 5 VPN 7 6 1 #2318 Switch 6 6 Signal 13 13 Flight 5 5 Launch 6 6 Info 19 18 1 #1409 Interface 14 14 Device 12 11 1 #2310 NetworkSel 6 3 3 #2846 Autoconnect 18 13 5 #2850 Advance 9 8 1 #1409 State 13 9 4 #2461 #2465 #2467,#1993,#1490 Total 204 183 21 >From test script Total PASS FAIL ToCheck TotalNumCase 127 123 4 0 connman 127 123 4 0 bt 14 12 2 0 #2514 ethernet 14 13 1 0 #2514 wifi 57 57 0 0 Guest 12 12 0 0 Functional 11 11 0 0 security 22 22 0 0 Advanced 6 6 0 0 Negative 6 6 0 0 signal 27 27 0 0 connman 14 14 0 0 manager 8 8 0 0 service 5 5 0 0 offline 0 0 0 0 profile 6 5 1 0 #2513 flight 15 15 0 0 Bug refered here is bug at bugs.meego.com, for example, bug #437 refer to http://bugs.meego.com/show_bug.cgi?id=437 For detailed test results, please see attached file Quality indicator ============================================== Feature Score Comments WiFi Buggy Bug 437, 610, 1578, 2312, 2640 Intel WiMAX N/A QA does not have WiMAX test environment Bluetooth PAN Good Bug 1378, 269, 2310, 3G data Buggy Bug 392, 1993, 1950, 2312, 2323, 2324 , 2503 Wired (LAN) Good State Bad Online/ready state is confusing: Bug 1490, 157, 1993, 2461, 2465, 2467, 2514 Network Selection Good Bug 2846 wpa-suplicant D-BUS Good profile management Good Bug 2513 manual switch connect Good Wired/WiFi co-exexist Good Remember device status Good connect to Broadcast AP Good connect to Hidden AP Buggy MeeGo bug 437, ConnMan needs to connect hidden service automatically Hidden Group AP Good auto connect reboot Buggy Bug 2850 auto connect Offline Good auto connect suspend Good favorite connection Good auto connectivity order Good WEP 40/128-bit key Good WEP 128-bit passphrase Good WPA personal Good WPA2 personal Good Open system Good Shared Key Good Remember passwd Good forgot password Good AP security detect Good auto del non-exist AP Good VPN Buggy Bug 2318 Plugin to oFono Good DNS Proxy Good DNS support via DHCP Good Static IP Good PPP 3G N/A WPA-enterprise Good Bests Jeff
ConnMan_3G/ NETWORK_3G_Coexist_001 pass NETWORK_3G_Coexist_002 pass NETWORK_3G_DeviceControl_001 pass NETWORK_3G_DeviceControl_002 pass NETWORK_3G_ICON_001 pass NETWORK_3G_Item_001 pass NETWORK_3G_Item_002 pass NETWORK_3G_Item_003 pass NETWORK_3G_Item_003 pass NETWORK_3G_Service_001 pass NETWORK_3G_Service_002 pass NETWORK_3G_Service_003 pass ConnMan_Advance/ CONNMAN_Advance_Hostname pass CONNMAN_Advance_ProfileCreate pass CONNMAN_Advance_ProfileName pass CONNMAN_Advance_StaticIP_BT pass CONNMAN_Advance_StaticIP_reboot pass CONNMAN_Advance_StaticIP_WIFI pass CONNMAN_Advance_StaticIP_Wired pass CONNMAN_profile_delete pass CONNMAN_Set_Nameservers_Negtive pass CONNMAN_Set_ Nameservers_Positivepass ConnMan_AutoConn/ CONNMAN_AutoConn_FavoriteSetting pass CONNMAN_AutoConn_LostConn pass CONNMAN_AutoConn_Reboot3G pass CONNMAN_AutoConn_RebootWIFI pass CONNMAN_AutoConn_RebootWired pass CONNMAN_AutoConn_RememNoSecu pass CONNMAN_Autoconn_RememWEP128 pass CONNMAN_AutoConn_RememWEP64 pass CONNMAN_Autoconn_RememWPAPSKAES pass CONNMAN_Autoconn_RememWPAPSKTKIP pass CONNMAN_Autoconn_HIDDEN_Rem NoSecubug2850 CONNMAN_Autoconn_HIDDEN_Rem WEP128bug2850 CONNMAN_Autoconn_HIDDEN_Rem WEP64bug2850 CONNMAN_Autoconn_HIDDEN_Rem WPAPSKAESbug2850 CONNMAN_Autoconn_HIDDEN_RemWPAPSKTKIP bug2850 CONNMAN_AutoConn_SuspendMem3G pass CONNMAN_AutoConn_SuspendMemWIFI pass CONNMAN_AutoConn_SuspendMemWired pass ConnMan_BT/ CONNMAN_BT_CheckProp_Powered pass CONNMAN_BT_Check_State pass CONNMAN_BT_Connect_Invalidnetwork pass CONNMAN_BT_Connect_PAN pass CONNMAN_BT_List_PAN pass ConnMan_Device/ CONNMAN_Device_3G pass CONNMAN_Device_3G_Conn pass CONNMAN_Device_3G_Reboot pass CONNMAN_Device_BT pass CONNMAN_Device_BT_Conn pass CONNMAN_Device_BT_Reboot fail(#2310) CONNMAN_Device_WIFI pass CONNMAN_Device_WIFI_Conn pass CONNMAN_Device_WIFI_Reboot pass CONNMAN_Device_Wired pass CONNMAN_Device_Wired_Conn pass CONNMAN_Device_Wired_Reboot pass ConnMan_Info/ CONNMAN_INFO_BtConnInfo pass CONNMAN_INFO_BTDevInfo pass CONNMAN_INFO_EthConnInfo pass CONNMAN_INFO_EthDevInfo pass CONNMAN_INFO_Exit_FlightMode pass CONNMAN_INFO_ManagerStateBTOffline pass CONNMAN_INFO_ManagerStateBTOnline pass CONNMAN_INFO_ManagerStateEthOffline pass CONNMAN_INFO_ManagerStateEthOnline pass CONNMAN_INFO_ManagerStateWiFiOffline pass CONNMAN_INFO_ManagerStateWiFiOnline pass CONNMAN_INFO_MultiConnection pass CONNMAN_INFO_NoConnList fail(#1490) CONNMAN_INFO_ProfileDefault pass CONNMAN_INFO_SetIPBT pass CONNMAN_INFO_TurnOn_FlightMode pass CONNMAN_INFO_WiFiConnInfo pass CONNMAN_INFO_WiFiDevInfo pass ConnMan_Launch/ CONNMAN_LAUNCH_boot pass CONNMAN_LAUNCH_BT pass CONNMAN_LAUNCH_dhclient pass CONNMAN_LAUNCH_OneConnmand pass CONNMAN_LAUNCH_reboot pass CONNMAN_LAUNCH_WpaSupplicant pass ConnMan_NetworkSel/ CONNMAN_NetworkSel_BroadcastNoSec pass CONNMAN_NetworkSel_BroadcastWEP pass CONNMAN_NetworkSel_BroadcastWPA pass CONNMAN_NetworkSel_HideNoSec fail(#2846) CONNMAN_NetworkSel_HideWEP fail(#2846) CONNMAN_NetworkSel_HideWPA fail(#2846) ConnMan_OfflineMode/ CONNMAN_Offline_Conn3G pass CONNMAN_Offline_ConnWIFI pass CONNMAN_Offline_ConnWired pass CONNMAN_Offline_Persistance pass CONNMAN_Offline_set pass ConnMan_Signal/ CONNMAN_SIGNAL_Change_Passphrase pass CONNMAN_SIGNAL_ChgEthPolicy pass CONNMAN_SIGNAL_DisableWiFiDev pass CONNMAN_SIGNAL_List_Networks pass CONNMAN_SIGNAL_MoveBT pass CONNMAN_SIGNAL_MoveWiFi pass CONNMAN_SIGNAL_MoveWiMAX pass CONNMAN_SIGNAL_RmvBT pass CONNMAN_SIGNAL_RmvEthCable pass CONNMAN_SIGNAL_SetEthPriority pass CONNMAN_SIGNAL_Set_Flightmode pass CONNMAN_SIGNAL_Set_WiFiNetworkMode pass CONNMAN_SIGNAL_StaticIPEth pass ConnMan_state/ CONNMAN_STATE_ETHENET_001 pass CONNMAN_STATE_ETHENET_002 pass CONNMAN_STATE_ETHENET_003 fail(#2465) CONNMAN_STATE_AvailableAP pass CONNMAN_STATE_IsolateAP fail(#2461) CONNMAN_STATE_Forbidden_service CONNMAN_STATE_3G fail(#1993) CONNMAN_STATE_System fail(#1490) ConnMan_Switch/ CONNMAN_Switch_EthToWiFi pass ConnMan_VPN/ CONNMAN_VPN_3G norun CONNMAN_VPN_Hidden pass CONNMAN_VPN_WEP pass CONNMAN_VPN_Wired pass CONNMAN_VPN_WPA pass CONNMAN_VPN_WPA2 pass CONNMAN_VPN_Disconnect fail(#2318) CONNMAN_VPN_Reconnect pass ConnMan_Wired/ CONNMAN_Wired_Conn pass CONNMAN_Wired_Hotplug pass WIFI/ConnMan_WiFi_CoExist/ CONNMAN_WIFI_CoExists_3G pass CONNMAN_WIFI_CoExists_BT pass CONNMAN_WIFI_CoExists_WIFI pass CONNMAN_WIFI_CoExists_Wired pass WIFI/ConnMan_WiFi_ConnectType/ CONNMAN_WIFI_Connect_NoSecuBroadcast pass CONNMAN_WIFI_Connect_NoSecuHide pass CONNMAN_WIFI_Connect_NoSecuHideGroup pass CONNMAN_WIFI_Connect_OpenWEP128 pass CONNMAN_WIFI_Connect_OpenWEP128Hide pass CONNMAN_WIFI_Connect_OpenWEP64 pass CONNMAN_WIFI_Connect_OpenWEP64Hide pass CONNMAN_WIFI_Connect_SharedWEP128 pass CONNMAN_WIFI_Connect_SharedWEP128Hide pass CONNMAN_WIFI_Connect_SharedWEP64 pass CONNMAN_WIFI_Connect_SharedWEP64Hide pass CONNMAN_WIFI_Connect_WPA2 pass CONNMAN_WIFI_Connect_WPA2Hide pass CONNMAN_WIFI_Connect_WPAEnterprise fail(#610) CONNMAN_WIFI_Connect_WPAPSKAES pass CONNMAN_WIFI_Connect_WPAPSKAESHide pass CONNMAN_WIFI_Connect_WPAPSKTKIP pass CONNMAN_WIFI_Connect_WPAPSKTKIPHide pass CONNMAN_WIFI_EnableDisable fail(#2640) WIFI/ConnMan_WiFi_Refresh/ CONNMAN_WIFI_Refresh pass CONNMAN_WiFi_Refresh_Disable pass CONNMAN_WiFi_Refresh_Enable pass WIFI/ConnMan_WiFi_Roaming/ CONNMAN_WIFI_Roaming_Broadcast fail(#?) CONNMAN_WIFI_Roaming_Hidden fail(#?) ConnMan_UI/3G/ NETWORK_3G_Coexist_001 pass NETWORK_3G_Coexist_002 pass NETWORK_3G_DeviceControl_001 pass NETWORK_3G_DeviceControl_002 pass NETWORK_3G_ICON_001 pass NETWORK_3G_Item_001 pass NETWORK_3G_Item_002 pass NETWORK_3G_Item_003 pass NETWORK_3G_Service_001 norun NETWORK_3G_Service_002 norun NETWORK_3G_Service_003 norun ConnMan_UI/BT/ NETWORK_BT_DeviceControl_001 pass NETWORK_BT_Item_001 pass NETWORK_BT_Item_002 pass NETWORK_BT_Item_003 pass NETWORK_BT_Item_004 pass NETWORK_BT_PANU pass ConnMan_UI/Carrick Hidden_Obsolete_Error pass Hidden_Unavail_Tech pass Too_Many_Notif pass Visible_Connecting_Network pass ConnMan_UI/ConfigAutoConnOrder/ NETWORK_profile_moveafter blocked#2247 NETWORK_profile_movebefore blocked#2247 ConnMan_UI/SaticIP/ NETWORK_StaticIP_BT pass NETWORK_StaticIP_reboot pass NETWORK_WIFI_StaticIP pass NETWORK_Wired_StaticIP pass ConnMan_UI/WIFI/ NETWORK_Coexist_WiFi_Wired_001 pass NETWORK_Coexist_WiFi_Wired_002 pass NETWORK_Coexist_WiFi_Wired_003 pass NETWORK_Coexist_WiFi_Wired_004 pass NETWORK_WIFI_setting_reboot pass NETWORK_WiFi_Switch_004 pass NETWORK_WiFi_Switch_005 pass NETWORK_WiFi_Switch_006 pass NETWORK_Wireless_005 pass NETWORK_Wireless_007 pass NETWORK_Wireless_008 pass NETWORK_Wireless_011 pass NETWORK_Wireless_012 pass NETWORK_Wireless_013 pass NETWORK_Wireless_014 pass NETWORK_Wireless_015 pass NETWORK_Wireless_016 pass NETWORK_Wireless_017 pass NETWORK_Wireless_017 pass NETWORK_Wireless_018 pass NETWORK_Wireless_019 pass NETWORK_Wireless_024 pass NETWORK_Wireless_025 pass NETWORK_Wireless_026 pass NETWORK_Wireless_027 pass NETWORK_Wireless_028 pass NETWORK_Wireless_029 pass NETWORK_Wireless_030 pass NETWORK_Wireless_031 pass NETWORK_Wireless_032 pass NETWORK_Wireless_033 pass NETWORK_Wireless_034 pass NETWORK_Wireless_035 pass NETWORK_Wireless_037 fail(#610) NETWORK_Wireless_038 pass NETWORK_Wireless_039 pass NETWORK_Wireless_80211 pass NETWORK_Wireless_80211x_hidden pass NETWORK_Wireless_HiddenSharedWEP128 pass NETWORK_Wireless_HiddenSharedWEP64 pass NETWORK_Wireless_SharedWEP128 pass NETWORK_Wireless_SharedWEP64 pass NETWORK_Wireless_UI_012 pass NETWORK_Wireless_UI_014 pass NETWORK_Wireless_WiFiSwitch_007 pass ConnMan_UI/Wired/ NETWORK_Ethernet_001 pass NETWORK_Ethernet_007 pass NETWORK_Ethernet_009 pass NETWORK_Ethernet_010 pass
==================== Test Report =================================== ====================================================================== Total PASS FAIL ToCheck TotalNumCase 127 123 4 0 connman 127 123 4 0 bt 14 12 2 0 ethernet 14 13 1 0 wifi 57 57 0 0 Guest 12 12 0 0 Functional 11 11 0 0 security 22 22 0 0 Advanced 6 6 0 0 Negative 6 6 0 0 signal 27 27 0 0 connman 14 14 0 0 manager 8 8 0 0 service 5 5 0 0 offline 0 0 0 0 profile 6 5 1 0 flight 15 15 0 0 ====================================================================== ********************************************************************** ********************************************************************** ********************************** bt ****************************** BigBroadcastPing.py PASS SmallBroadcastPing.py PASS HasIP.py PASS IsDHCP.py PASS IsPoweredOn.py PASS Disconnect.py PASS Connect.py PASS IsReady.py PASS DeviceIsExist.py PASS PoweredOff.py PASS PoweredOn.py PASS IsReEnableWorks.py PASS ifconfig_wifi_down.py FAIL #2514 ifconfig_wifi_up.py FAIL #2514 Test report for connman/bt CountNumCase 14 PASS 12 FAIL 2 ToCheck 0 ********************************** ethernet ****************************** BigBroadcastPing.py PASS HasIP.py PASS Connect.py PASS DeviceIsExist.py PASS Disconnect.py PASS IfconfigDown.py FAIL #2514 IfconfigUp.py PASS IsDHCP.py PASS IsPoweredOn.py PASS IsReady.py PASS IsReEnableWorks.py PASS PoweredOff.py PASS PoweredOn.py PASS SmallBroadcastPing.py PASS Test report for connman/ethernet CountNumCase 14 PASS 13 FAIL 1 ToCheck 0 ********************************** Guest ****************************** BigBroadcastPing.py PASS SmallBroadcastPing.py PASS HasIP.py PASS IsDHCP.py PASS IsPoweredOn.py PASS Disconnect.py PASS Connect.py PASS IsReady.py PASS DeviceIsExist.py PASS PoweredOff.py PASS PoweredOn.py PASS IsReEnableWorks.py PASS Test report for connman/wifi/Guest CountNumCase 12 PASS 12 FAIL 0 ToCheck 0 ********************************** Functional ****************************** Channel10.py PASS Channel11.py PASS Channel1.py PASS Channel2.py PASS Channel3.py PASS Channel4.py PASS Channel5.py PASS Channel6.py PASS Channel7.py PASS Channel8.py PASS Channel9.py PASS Test report for connman/wifi/Functional CountNumCase 11 PASS 11 FAIL 0 ToCheck 0 ********************************** security ****************************** Broadcast.py PASS Hidden.py PASS OpenBroadcastRSN.py PASS OpenBroadcastTKIP.py PASS OpenBroadcastWEP128.py PASS OpenBroadcastWEP40.py PASS OpenHiddenRSN.py PASS OpenHiddenTKIP.py PASS OpenHiddenWEP128.py PASS OpenHiddenWEP40.py PASS Open.py PASS PSK1.py PASS PSK2.py PASS RSN.py PASS WEP128.py PASS WEP40.py PASS WPA.py PASS SharedBroadcastWEP128.py PASS SharedBroadcastWEP40.py PASS SharedHiddenWEP128.py PASS SharedHiddenWEP40.py PASS Shared.py PASS Test report for connman/wifi/security CountNumCase 22 PASS 22 FAIL 0 ToCheck 0 ********************************** Advanced ****************************** AutoConnectFromSuspend.py PASS AutoDelete.py PASS DetectSecurity.py PASS Disconnect.py PASS ForgotPassword.py PASS RememberPasswd.py PASS Test report for connman/wifi/Advanced CountNumCase 6 PASS 6 FAIL 0 ToCheck 0 ********************************** Negative ****************************** NegWEP128ascii.py PASS NegWEP128Long.py PASS NegWEP128Short.py PASS NegWEP40ascii.py PASS NegWEP40Long.py PASS NegWEP40Short.py PASS Test report for connman/wifi/Negative CountNumCase 6 PASS 6 FAIL 0 ToCheck 0 ********************************** connman ****************************** EnabledTechnologies.py PASS Networks.py PASS Powered.py PASS Scanning.py PASS PSK2.py PASS Services.py PASS RSN.py PASS State.py PASS Connected.py PASS ConnectedTechnologies.py PASS DefaultTechnology.py PASS Favorite.py PASS IPv4.py PASS OfflineMode.py PASS Test report for connman/signal/connman CountNumCase 14 PASS 14 FAIL 0 ToCheck 0 ********************************** manager ****************************** ConnectedTechnologies.py PASS WEP128.py PASS DefaultTechnology.py PASS EnabledTechnologies.py PASS Profiles.py PASS Services.py PASS State.py PASS Strength.py PASS Test report for connman/signal/manager CountNumCase 8 PASS 8 FAIL 0 ToCheck 0 ********************************** service ****************************** AutoConnect.py PASS Favorite.py PASS WEP40.py PASS IPv4.py PASS Passphrase.py PASS Test report for connman/signal/service CountNumCase 5 PASS 5 FAIL 0 ToCheck 0 ********************************** offline ****************************** Test report for connman/signal/offline CountNumCase 0 PASS 0 FAIL 0 ToCheck 0 ********************************** profile ****************************** Add.py PASS AddExist.py PASS List.py PASS Remove.py PASS RemoveNoExist.py PASS Test report for connman/profile CountNumCase 5 PASS 5 FAIL 0 ToCheck 0 ********************************** flight ****************************** WiFiGuest.py PASS Offline.py PASS Service.py PASS EthNoPing.py PASS EthIsPoweredOff.py PASS WiFiNoPing.py PASS WiFiIsPoweredOff.py PASS EthPoweredOn.py PASS WPA.py PASS EthPoweredOff.py PASS WiFiPoweredOn.py PASS WiFiPoweredOff.py PASS Online.py PASS WiFiOnline.py PASS EthOnline.py PASS Test report for connman/flight CountNumCase 15 PASS 15 FAIL 0 ToCheck 0 ====================================================================== ====================================================================== Test report for the whole test suite TotalNumCase 127 PASS 123 FAIL 4 ToCheck 0 ToCheck
_______________________________________________ connman mailing list connman@connman.net http://lists.connman.net/listinfo/connman