Summary

eZuce is pleased to announce the Release of uniteme 18.04.

Dev & QA teams at eZuce have been busy as usual. We put the next generation work on hold to ensure growth to 30,000+ users for one of our large customers. The biggest change is related to Mongo. MongoDB has been upgraded to version 3.4. This has the advantage of running Mongo’s WiredTiger Storage Engine. This brings with it document level locking vs. the entire collection as in MongoDB version 2.6. Additionally, database sizes will grow and shrink according to what data is stored in the database (vs. just growing forever).

In 18.04 you’ll also find some LDAP improvements, several new REST improvements and some performance tweaks.

IMPORTANT: We’re instituting a new method to upgrade so please read the complete release notes!!

Reachme is now a standalone product and is not included in 18.04 release. If you use Reachme on 17.10 do not upgrade to 18.04. Reachme V3 Beta is has been announced. If you’d like to evaluate it please contact sa@ezuce.com.

Also, make sure you see Note #2 below regarding disk space and Note #3 for new upgrade instructions!

We’re back at the next-gen work now. Our main focus for 18.08 will be operating on CentOS 7 and allowing for the use of flat file storage for storing voicemail (for those wanting to use mounted storage from a distributed file system or highly available storage). We’re still pushing ahead with Docker Containers as well but aren’t expecting that work will make it into 18.08.

Highlights

uniteme New Features:

  • MongoDB Version 3.4

uniteme Improvements:

  • sipDiagram Builder Enhancements

  • Download entire CDR reports

  • Several REST API Improvements

  • LDAP Username import manipulation - Strip, Append, Prepend

  • LDAP Import Settings now per LDAP Server

  • Proxy Statistics Manager enhancements to provide additional statistics

  • Make EntityDB cache expiration configurable

  • Mark SELinux as enforcing and add Management to Admin GUI

  • Performance related change that re-routes SIP Subscription Traffic to bypass Registrar

  • Performance option to disable Proxy DNS Lookups

Notes

  1. Full Release Notes with installation information are located here: 18.04 Full Release Notes

  2. Please note that the upgrade will take extra disk space. The MongoDB upgrade essentially makes a copy of the existing database and then re-mounts it. Please be prepared for this.

  3. Please also note that if you have a multi-server cluster vs. a single server there are specific upgrade instructions.

  4. Also, note UC-4721 regarding backup and restore.

  5. We're still working on the ISO, the new version of Mongo is giving us some difficulty here. Please use the RPM method of installation for fresh installs for now.

Who Should Install?

This release is recommended for all 4.6 and later installations except for customers who are running Reachme. If you have a patch installed to your system a new patch may be required. Please contact sa@ezuce.com if think you may have a patch applied as that may be replaced during the update.

eZuce's software products continuously progress through an Agile based development methodology that keeps feature functionality comprehensive and up-to-date in response to evolving market and customer requirements.

New software releases are made at a rate of two to four releases a year. Releases are numbered in the <yy>.<mm>.<uu> format where <yy> and <mm> designate the year and the month, respectively, in which a release is made generally available. Where applicable, <uu> corresponds to an update release relative to a general release on which fixes are made available.

In order to ensure service continuity and stability, customers may keep their production environments unchanged for up to a 6-month period during which release updates or patches are made available. After a release is more than 6-months old, eZuce customers would have to upgrade to the latest generally available release - inclusive of all fixes to date and any new patches.

Questions

If you have questions about updating you can email sa@ezuce.com or if you need assistance with the update contact your account manager or email sales@ezuce.com.

Software Release History

We're currently running on a 4-month release cycle.

  • April release for 2018 is 18.04
  • August release for 2018 is 18.08
  • ...

 

Release Level History

  • 14.04   - April 30, 2014
  • 14.04.1 - June 01, 2014
  • 14.04.2 - July 11, 2014
  • 14.04.3 - October 24, 2014
  • 14.10 - February 5, 2015
  • 15.04 - April 29, 2015
  • 15.05 - May 27, 2015
  • 15.06 - June 30, 2015
  • 15.08 - August 31, 2015
  • 15.10.1 - December 9, 2015
  • 15.12 - January 6, 2015
  • 16.02 - March 14, 2016
  • 16.04 - May 31, 2016
  • 16.08 - October 6, 2016
  • 16.12 - January 17, 2016
  • 17.04 - April 18, 2017
  • 17.08 - September 7, 2017
  • 17.10 - December 7, 2017
  • 18.04 - June 5, 2018

System Requirements

For a reasonably performing system, we recommend the following configuration.

Minimum hardware requirements

  • Pentium 4 or Xeon processor @ 2.0 GHz Core 64bit or higher
  • Minimum 8 GB of RAM with sufficient swap space
  • 80 GB disk (75 users depending on usage patterns)

Notes:

  • uniteme supports an unlimited number of voicemail boxes, the total number of hours of recorded messages is determined by the size of the hard-disk. As a rule, for every minute of recorded messages, you will need 1 MB of disk space (About 3 hours per 10 GB of disk space).

Operating System

CentOS/RHEL 6 x86_64 with latest updates is required.

Devices

Phones

  • Polycom VVX Devices with firmware 5.5.2 (split) are recommended for new installations
  • Polycom SoundPoint IP Devices should run firmware 4.0.11 (split)

Gateways

  • AudioCodes Gateways are recommended for PSTN connectivity

SBCs

  • Frafos, Sangoma, Acme Packet and Ingate SBC's are recommended for SIP Trunking and Remote Worker connectivity (commonly referred to as sipXbridge and MediaRelay services respectively).
  • NOTE: The eZuce uniteme - "Use built-in SIP Trunk SBC" found in Gateway Details for use with Trunking or Remote Worker solutions should be used only for lab purposes. The uniteme "Built-In SIP Trunk SBC" (sipXbridge) will not be supported in any production or live environment. Additionally, sipXbridge does not work in an HA environment.

Documentation

Technical Reference Manuals, User Guides, Reach Reference Manuals, and other technical and user information can be found under the following link: Documentation Page

Installation and Upgrade Notes

Special MongoDB note

Please be aware of these Mongodbrequirements http://docs.mongodb.org/manual/reference/ulimit/ Note: Both the “hard” and the “soft” ulimit affect MongoDB’s performance. The “hard” ulimit refers to the maximum number of processes that a user can have active at any time. This is the ceiling: no non-root process can increase the “hard” ulimit. In contrast, the “soft” ulimit is the limit that is actually enforced for a session or process, but any process can increase it up to “hard” ulimit maximum.Every deployment may have unique requirements and settings; however, the following thresholds and settings are particularly important for mongod and mongos deployments:

ulimit –a
-f (file size): unlimited
-t (cpu time): unlimited
-v (virtual memory): unlimited
-n (open files): 64000
-m (memory size): unlimited
-u (processes/threads): 32000

 

Always remember to restart your mongod and mongos instances after changing the ulimit settings to make sure that the settings change takes effect.If you limit virtual or resident memory size on a system running MongoDB the operating system will refuse to honor additional allocation requests. After every install/upgrade please check that "cat /proc/$pid_of_mongo/limits" have the recommended value of 655350. To make this value permanent you need to create this file /etc/security/limits.d/99-mongodb-nproc.conf and add the following lines:

mongodb soft nproc 64000
mongodb hard nproc 64000
mongodb soft nofile 64000
mongodb hard nofile 64000

Special Patch Note

If you have a patch installed to your system a new patch may be required. Please contact sa@ezuce.com if think you may have a patch applied as that may be replaced during the update.

Installing from ISO image

Download uniteme ISO

Download the ISO image corresponding to your hardware and write the image to a DVD.

Install uniteme

  • Boot from the DVD created with the uniteme ISO image.
  • Press Enter at the boot screen below to begin the uniteme installation.
  • Select Manual Configuration under Enable IPv4 support and select OK.
  • Set a static IPv4 address with the corresponding networking information and click OK.
  • In certain situations, a warning of the use of indicated storage devices will be displayed.
  • Select the language to be used during the installation.
  • Select the keyboard layout to be used.
  • Select the timezone to be used.
  • Set a root password.
  • Login to the system as root with the password you provided earlier and continue on to the Configure of uniteme.

Installing from Repository

uniteme can be installed using the following procedure

  • Download and install CentOS 6.x minimal ISO
  • Run the following command:
yum update && reboot
  • Run the following commands to retrieve and run the eZuce uniteme installer:
curl https://download.ezuce.com/openuc-setup > /usr/bin/openuc-setup
chmod +x /usr/bin/openuc-setup
openuc-setup

This utility will guide you through the process of installing uniteme from the eZuce software repository.

Upgrade from previous versions

New Procedure

We will be utilizing an upgrade script to ensure upgrades proceed as intended and so that customers have the appropriate warnings and information before upgrading.

Warning!

Make sure you backup your system (configuration and voicemail at a minimum) prior to installation. You'll be upgrading mongodb to a new version!

Upgrade to 18.04 for Single Server systems

Backup your System

Login to the Admin GUI and click on System -> Backup and at a minimum backup configuration and voicemail.

Do it now... before you go any further.

Download upgrade.sh

Login to the primary server as root.

Execute the following:

wget https://'USERNAME:PASSWORD'@download.ezuce.com/openuc/upgrade.sh
chmod +x upgrade.sh

Run upgrade.sh

Execute the upgrade script and answer 'Yes' to continue:

./upgrade.sh

 

The following will be displayed:

Uniteme 18.04 Upgrade Script
 
IMPORTANT: If this is a multi-server cluster, all databases except the Primary (which must be on the configuration server) should be removed.
IMPORTANT: You should run a system backup and copy your config and voicemail backups to another system. If the upgrade fails, you will need to build a new server and restore from backup.
IMPORTANT: Ensure that you have enough disk space available for a copy of the Mongo databases. (roughly your Config + Voicemail backups).
IMPORTANT: 18.04 does not have Reachme in it, if you use Reachme on this server or in the cluster, do not continue!
 
This script will do the following:
- Back up mongo config and dbs
- Stop mongo instance and remove old mongo files
- Change 17.10 occurrences in /etc/yum.repos.d with 18.04
- Perform yum update and then reboot the machine
 
On sipxconfig service startup following steps are taken (in case there is a backup still on the disk):
- Restore mongo config and dbs, then remove from disk
- Reboot machine
 
For other cluster servers:
- Run same script
- Re-add databases that were removed.
 
Continue? (you must enter Yes or No as shown and press Enter):

Enter 'Yes' and press Enter to continue. The system will reboot a couple of times as part of the process.

Upgrade to 18.04 for Multi-Server Clusters with Multiple Database Servers

Backup your System

Login to the Admin GUI and click on System -> Backup and at a minimum backup configuration and voicemail.

Do it now... before you go any further.

Remove All Secondary Databases from Cluster Nodes

From the Admin UI remove databases from all nodes except for the primary node. Click on System -> Databases do accomplish this.

Proceed as with Single Server for Primary

Proceed as above with the 'Download upgrade.sh' section and then the 'Run upgrade.sh' section.

Secondary Servers in Cluster

After the Primary Server has completed, repeat for each of the Secondary Servers in the cluster until all are completed. 

Add Database Nodes

After the secondary nodes are complete and done with their reboots, login to the Admin UI and add back the database nodes that were removed.

Send Server Profiles

Login to web interface as superadmin.

Navigate to System -> Servers page. Place check mark next to server names and click 'Send Profiles'.

SEC Service Upgrade Note

When upgrading uniteme from openUC 4.6 Update 11 or 14.4.3 to 15.06 follow these steps to ensure the SEC service is correctly running:

  • 1. Upgrade from 4.6 Update 11 or 14.4.3 to 15.06.
  • 2. After the upgrade is complete, perform the usual restart.
  • 3. Once possible, connect via CLI and monitor processes using top. Notice that the SEC process is using a lot of CPU memory.
  • 4. Perform another restart OR restart only the Sipxlogwatcher service.

Modified Files Upgrade Note

If you have manually modified any system related files or some files are not as yum would expect them to be, the yum update process may not overwrite them. It will instead create 'rpmnew' or 'rpmsave' files and not overwrite the files. The administrator may have previously modified the files knowingly or as part of a patch supplied by TAC.

To check your upgrade.log and search for *.rpmnew *.rpmsave on your system check the upgrade log:

You will be responsible for merging any changes from the old file to the new or contacting Technical Support if you require assistance.

Support Tips and Contact Information 

Please see the Getting Support section for support tips and support contact information

Issues Sorted by Issue Number

 JIRA nameRN ContentEnhancement/Fix/Known IssueKeywords
SIPX-695LDAP improvementCurrently "Username strip" under LDAP management applies to all LDAP imports. An administrator would like for this to be associated with each LDAP server.EnhancementLDAP
SIPX-707Upgrade MongoDB to 3.4 (or 3.6 if available)Why:
- WiredTiger storage, no allocation of disk
- no more collection global lock but document lock (should greatly improve performance, get benchmarks from older versions vs 18.04)
- better usage of CPU cores
- up to 50 nodes in replica set

ToDo:
- provide latest Mongodb RPMs on sipXcom install
- rework sipxecs-setup procedure
- adapt database management from sipXconfig
- adapt backup/restore procedures
- adapt local regions feature
- test basic functionality
- full regression testing & bug fixing
EnhancementMongo
SIPX-713Add Phone Management to REST APIThis is a request to add the ManagePhone functions that are currently in the SOAP API to the REST API. Specifically the generateProfiles and restart capabilities, although all of them would be nice.

Examples:
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/user/id/10/sendProfile
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/user/id/10/sendProfile/restart
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/user/name/200/sendProfile
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/user/name/200/sendProfile/restart
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/phoneGroup/id/18/sendProfile
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/phoneGroup/id/18/sendProfile/restart
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/phoneGroup/name/phgr1/sendProfile
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/phoneGroup/name/phgr1/sendProfile/restart
EnhancementAPI
SIPX-717CallQueue Default for RecordingChange CallQueue Default for Recording to Disabled.

At present default is Enabled and recording directory is blank which can cause a problem.
EnhancementCallQueue
SIPX-719Add search phone based on user / line to REST APIAdd ability to retrieve phones for an user / line through REST APIEnhancementAPI
SIPX-720REST API - Add ability to search users based on emailImprove users rest api to retrieve users based on email field.EnhancementAPI
SIPX-721Add option to disable Proxy DNS lookupsThis is a performance tweak that can make a big difference with the performance of very large systems.

- Add option in UI needed to disable registrar and MWI lookups from proxy
- In case option checked then generate forwardingrules.xml with IP:port instead DNS record. For Registrar that should be easy as proxy and registrar are automatically enabled on same node. Figure out IP address to point to for MWI server.
- By default, option is disabled
EnhancementProxy
SIPX-722Registrar fails to start when 400k subscription in mongo databaseOn startup all components using SipSubscribeServer try to retrieve and load in memory all entries within database.

In case of SIPX-722 there were too many records to fit in returned cursor so that one failed with

"2018-01-31T11:00:03.256938Z":22:SIP:ERR:int.phx2.ezuce.ro:SipSubscribeServer-29:7f5b7de25700:sipxregistry:"SipSubscribeServer::initialize failed - retrying after 500 milliseconds"
"2018-01-31T11:00:03.860614Z":23:SIP:EMERG:int.phx2.ezuce.ro:SipSubscribeServer-29:7f5b7de25700:sipxregistry:"ALARM_MONGODB_SLOW_READ Last Mongo read took a long time: document: node.subscription delay: 103 milliseconds"

This caused no registrars to be able to start. This is an edge case that exposed the bug, however, using such approach of loading all subscribes from database when process starts (and assuming number of records can fit in mongo cursor) could in normal operation result in:
- slow operating of mongo database (restarting a process or trying to recover by restarting processes at the same time can nail down entire cluster)
- slow start of SIP processes using this approach (checking the code it seems like not only registrar is using same approach, but also MWI, proxy, RLS and SAA)

The query that used to return all registrations from MongoDB (see below) was removed
FixRegistrar
SIPX-723Add possibility to specify urls in product copyright valueAn administrator would like to customizing skin (as documented in: http://wiki.sipxcom.org/display/sipXcom/Customizing+Colors%2C+Layout+and+Logo)

The administrator would like to allow the inserting of links in product.copyright value.
EnhancementPortal
SIPX-725Mark selinux as enforcing.Currently /etc/selinux/config is disabled.

Some customers may require it to be enforcing for security reasons.
EnhancementSystem
SIPX-726Create proper compound indexes for subscriptions and registrar databasesDuring latest load / profiling it was noticed that some operations were reported slowly.

- for updating registrations bindings
- for updating subscriptions bindings

Comparing queries with collection indexes it can be seen that following fileds that are queried were not indexed:
- for registrar: contact, shardid
- for subscription: fromUri, callId, eventTypeKey

Used compound indexes, see https://docs.mongodb.com/manual/core/index-compound/
EnhancementRegistrar Mongo
SIPX-727Update registration bindings using upsert instead delete / insertUpdating registrations was made up from two operations: delete registration then update registration.
Subscription database is already using upsert call instead remove / update, change database for updating registrations:
https://github.com/sipXcom/sipxecs/blob/release-18.04/sipXcommserverLib/src/sipdb/SubscribeDB.cpp#L174
EnhancementRegistrar Mongo
SIPX-728Call ensureIndex only once for registrations and subscribtions databasesEach time a registration is updated / inserted, a call to ensure database indexes is performed. This could be resource consuming and affecting system performance.

More, starting with version 3.0 this call is deprecated and it uses createIndexes call behind the scene, see: https://docs.mongodb.com/manual/reference/method/db.collection.ensureIndex/#db.collection.ensureIndex

Remove ensureIndex from CRUD operations and keep it only when service starts up.
EnhancementRegistrar Mongo
UC-4415Openuc-saa service requires a restart after server rebootOpenUC-saa service does not work properly (presence not working) on fresh server start or after reboot, until service restart is issued.

Added back a restart of saa service after 5 minutes of server run time.
FixSAA
UC-4549SipDiagram Builder: Create archive to view diagrams offlineAn administrator would like to be able to view SipDiagram Builder related information offline. Create an email-ready zip archive with all required files to be able view diagram offline.

The files are generated in the location /var/www/html/siplogviewer/tempdata/CALL_ID.
EnhancementDiagnostics
UC-4622Administrator DashboardAn Administrator would like to be able to see performance information about the servers operating in their Uniteme system.

sipXproxy Performance Information - SIP Messages per second, Queue Depth, Auth Cache Queue entries
sipregistrar Performance Information - # Registrations, # Registration requests per second
MongoDB Performance Information - reads / sec, writes / sec, etc.
System Performance - Packets / sec, CPU use, memory use, etc.

Display last X messages from important log files.

It's not required to build this dashboard in the current admin gui.
EnhancementDiagnostics
UC-4627REST API improvement to change group firmware for phone groupsA user would like to change group firmware for phone groups via REST calls.EnhancementAPI
UC-4637Re-route SIP Subscription TrafficAt the moment all SUBSCRIBE messages which are not directed to MWI are explicitly sent to Registrar by default.
Registrar the forwards subscriptions to SSS using 302. This creates unnecessary traffic load on Registrar and underlying the mongo database.

We will change forwardingrules.xml in a way to forward subscriptions to SSS by default without making another hop with Registrar.

Forwarding rules part:
<!-- All other SUBSCRIBE requests go to the SIP registry service -->
<routeTo ruriParams="sipx-noroute=Voicemail">&lt;rr.HOSTNAME:0;transport=tcp;x-sipx-routetoreg&gt;</routeTo>

should be changed to
<!-- All other SUBSCRIBE requests go to the SSS service -->
<routeTo>&lt;HOSTNAME:5140;transport=tcp&gt;</routeTo>

where HOSTNAME is IP of SSS and 5140 is SSS default port
EnhancementRegistrar
UC-4640Make EntityDB cache expiration configurable (sip code part)The Proxy Service already has a caching mechanism already implemented. With this Jira we will implement a flexible way to configure the cache timeout used for packets received by the proxy without digest (meaning with no authorization yet). Today, the cache is hardcoded to 30 seconds but it can be set to several minutes depending on the configuration required.

Also to avoid a possible synchronization issue between an admin deleting a user and having the UA not yet removed from the cache (even it is not happening often), we may also consider a simple API call so that when the Administrator performs a delete action, we delete the associated entry in the cache (or maybe a special "SIP delete" message to proxy for example).

Additionally, run tests at various traffic levels starting at 30 seconds to 3 minutes in 30-second intervals, 5 minutes and 10 minutes to pattern behavior and find optimum setting.
EnhancementProxy
UC-4641Make EntityDB cache expiration configurable (webui part)Add tosipxproxy config file (/etc/sipxpbx/sipXproxy-config): name: SIPX_PROXY_ENTITY_CACHE_EXPIRE, default is 30.
Add to sipxregistrar config file (/etc/sipxpbx/registrar-config) : name: SIP_REGISTRAR_ENTITY_CACHE_EXPIRE, default is 30.

Actually this should be single setting "Entity Cache Expiration" on webui, but value should be extracted into 2 files.
EnhancementProxy
UC-4644Add additional data to Proxy Statistics ManagerExtend the new StatisticsManager service with additional metrics we need. Make all information available in the existing .json file.

Count the different SIP Methods, SIP Error Responses and to who, Top SIP 10 talkers overall, and additional information that would be useful for performance monitoring (tbd).

{"timestamp": "2018-02-16T15:50:54.088061Z", "REGISTERi": 110}
{"timestamp": "2018-02-16T15:50:54.088076Z", "REGISTERiF": 110}
{"timestamp": "2018-02-16T15:50:54.088091Z", "REGISTERiS": 0}
{"timestamp": "2018-02-16T15:50:54.088101Z", "REGISTERo": 375}
{"timestamp": "2018-02-16T15:50:54.088111Z", "REGISTERoF": 110}
{"timestamp": "2018-02-16T15:50:54.088120Z", "REGISTERoS": 0}
{"timestamp": "2018-02-16T15:50:54.088129Z", "proxy_active_transaction_count": 5255}
{"timestamp": "2018-02-16T15:50:54.088138Z", "proxy_avg_dispatch_speed": 0}
{"timestamp": "2018-02-16T15:50:54.088158Z", "proxy_avg_entity_db_read": 0}
{"timestamp": "2018-02-16T15:50:54.088167Z", "proxy_avg_regdb_read": 0}
{"timestamp": "2018-02-16T15:50:54.088177Z", "proxy_msq_queue_size": 0}
{"timestamp": "2018-02-16T15:50:54.088186Z", "proxy_ua_queue_size": 0}
{"timestamp": "2018-02-16T15:50:54.088195Z", "queue_size": 93}
{"timestamp": "2018-02-16T15:50:54.088204Z", "reqi": 220}
{"timestamp": "2018-02-16T15:50:54.088213Z", "reqo": 1117}
{"timestamp": "2018-02-16T15:50:54.088222Z", "rspi": 247}
{"timestamp": "2018-02-16T15:50:54.088231Z", "rspo": 383}
{"timestamp": "2018-02-16T15:51:00.039017Z", "BYEi": 130}
{"timestamp": "2018-02-16T15:51:00.039045Z", "BYEiF": 109}
{"timestamp": "2018-02-16T15:51:00.039064Z", "BYEiS": 0}
{"timestamp": "2018-02-16T15:51:00.039075Z", "BYEo": 706}
{"timestamp": "2018-02-16T15:51:00.039084Z", "BYEoF": 109}
{"timestamp": "2018-02-16T15:51:00.039097Z", "BYEoS": 0}

Please note that the following metrics are presently constantly 0 : proxy_avg_dispatch_speed, proxy_avg_entity_db_read, proxy_avg_regdb_read, proxy_msq_queue_size, proxy_ua_queue_size.

New Proxy Statistics Manager can be disabled for higher performance. There is a small performance penalty for writing these statistics to the .json file.

EnhancementDiagnostics
UC-4645LDAP Username import manipulationAn administrator would like more flexibility to manipulate usernames during LDAP import.

1. Add the ability to strip characters from the imported username via regex
2. Add the ability to append a suffix to the imported username
3. Add the ability to prepend a prefix to the imported username
EnhancementLDAP
UC-4660Sipxsupervisor enables nfs services alwaysIt seems that by default /usr/share/sipxecs/cfinputs/plugin.d/sipxtcpdumplog.cf enables nfs services by default, even if the tcpdump services are disabled in sipxconfig webui. This leaves the nfs rpc related services running and bound to ports.

As a workaround the administrator could disable rpcbind and rpcgssd services, moved sipxtcpdumplog.cf out of the plugin.d directory, and rebooted.
FixSystem
UC-4665Remove Reachme from UnitemeWith Reachme moving to a standalone product at the end of May we will remove Reachme from Uniteme installation.

This will require that we address removing Uniteme and components during upgrades. We'll also address any remove backup tasks, etc.
EnhancementSystem
UC-4676Fix misleading log message queue is over half fullDuring investigations it turned out that this log is wrong and "max - 100" was a hardcodded and unused value. Determine and log proper message with correct message queue depth.

2018-01-24T13:23:27.343137Z":122375:KERNEL:WARNING:s1.phx2.ezuce.ro:SipClientTcp-21:7fb4a8089700:sipxproxy:"OsMsgQShared::doSendCore message queue 'SipClientTcp-20' is over half full - count = 6818 max = 100"
FixProxy
UC-4677Misleading mongodb slow read alarmsWhen the primary server was shutdown (so no mongo for processes to connect to) proxies on running nodes were given ALARM_MONGODB_SLOW_READ and some exact time, e.g. 814 ms. So this is not something reported by mongo (as a reminder mongodb already logs slow queries in mongodb.log) and there is the possibility that such alarms to be wrongly raised (as a consequence of slowly processing for example).

- Double check code and correct scenarios where these slow read reports could be wrongly reported
FixProxy
UC-4684Call queue MoH replaces default MoH

Uploading CallQueue MoH seems to override the system moh setting. Navigating to features -> moh and applying default.wav works to correct, however after the default.wav plays the call queue MoH is next in line.


For existing queues customer must: 
1. Move all queue audio files from /var/sipxdata/mediaserver/data/moh to /var/sipxdata/mediaserver/data/prompts (use mv comand) 
or 
2. Upload the Queue Audio files again in UI 
and then manually delete old files from /var/sipxdata/mediaserver/data/moh

FixCallQueue
UC-4686REST API improvement: change multiple settings with one rest callA customer would like to edit phone groups more efficiently via API.

A single setting for a phone group can be performed. For example:
https://localhost/sipxconfig/api/phoneGroups/34/model/polycom450/settings/lcl/time/24HourClock
with data 1 or 0 and it will update that one setting. What would be helpful would be to update multiple settings in a single API call.

Now to change multiple phone group settings in a call:
curl -X PUT --insecure -H "Content-Type: application/json" -d '{"settings":[{"path":"lcl/time/24HourClock","value":"0"},{"path":"lcl/datetime/date.format","value":"Md,D XX"}]}' https://superadmin:11111111@localhost/sipxconfig/api/phoneGroups/phgr1/model/polycom330/settings
EnhancementAPI
UC-4687Download entire CDR reports (currently limited to 5MB)An administrator is trying to pull yearly CDR reports via the historic page. No matter what time range is set once it is downloaded file is always MAX 5MB and only shows few days.EnhancementCDR
UC-4695When generating very large CSV history or reports, web interface crashesThis has occured on a server containing ~2.000.000 calls in history. When trying to generate either a history CSV either a report with all the calls, the client browser receives error 502 - Proxy Error. However, the process is still running in the background taking up resources, generating a file that will not be able to be downloaded. The web GUI cannot be contacted anymore and sipxconfig will need a restart.

This would definitely be a problem on single node, or if the proxy, registrar and other services are running on the same node.

The system the error occured in is on AWS and contains dual Xeon processors and 8 GB of ram. The system generates a csv with up to 1.000.000 entries in around 20 seconds.

25.000 entries - 4.3 MB
200.000 entries - 35 MB
300.000 entries - 52 MB
500.000 entries - 88 MB
1.000.000 entries - 174 MB
Over 1.100.000 entries the system cannot be contacted anymore.

The same problem goes for reports as well, system stops responding when you try to generate a report from a time period that has more than 2.000.000 entries.
FixCDR
UC-4699Openuc-saa and sipxsss tail oplog using $or queryWhile profiling mongo database some slow queries were noticed.

Deactivating openuc-saa and sipxsss improves load testing results.

These slow queries are triggered by openuc-saa / sipxsss tailing oplog for getting updates when an user is changed (speed dial with presence added / deleted, shared user activated / deactivated). Problem was that openuc-saa / sipxsss mechanism scans the entire oplog (that can have millions of documents).

Change saa and sss to subscribe to a specific namespace (that should be the imdb database) to improve performance.
EnhancementSSS SAA
UC-4706SIP Status memory consumption increases infinitelyA memory leak was discovered in sip status - when stressing the system with many subscribes, SIP status keeps increasing in memory consumption. On the test system the consumption of SIP status increased to over 40% of the total memory (8 GB Ram) and system killed the mongod process to free space.FixSIPStatus
UC-4721When restoring a backup made on a previous version on another cluster, the servers are not correctly added

A bug has been verified in 17.10 and 18.04 when trying to restore a backup from a previous version of code.

To reproduce -

  1. Create a backup on cluster 1 running version 16.12 of UniteMe.
  2. Restore the backup on cluster 2 running version 18.04 of UniteMe. When doing this, the system -> servers primary is correctly kept, but the servers 2 and 3 are replaced with the ones in 16.12 and hence loose connection becoming uninitialized. Also, system -> databases does not load anymore. Checking rs.status() shows correctly the databases on the nodes before the restore. Also, modifying the domain and IP of the secondary nodes to the correct ones does not work.

WORKAROUND:
1. Prepare the 18.04 system for restoring from 16.12. Delete the secondary and arbiter databases.
2. Restore the config file. After the restore process is complete, the secondary servers will be replaced by the one on the original system.
3. Delete the secondary servers.
4. Re-add the secondary servers. sipxecs-setup to reset the servers and re-assign them with the new IDs provided.
5. After the servers are configured, go to system -> databases and add the secondaries to the system.

Known IssueBackup

 

Issues Sorted by Keyword

 JIRA nameRN ContentEnhancement/Fix/Known IssueKeywords
SIPX-713Add Phone Management to REST APIThis is a request to add the ManagePhone functions that are currently in the SOAP API to the REST API. Specifically the generateProfiles and restart capabilities, although all of them would be nice.

Examples:
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/user/id/10/sendProfile
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/user/id/10/sendProfile/restart
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/user/name/200/sendProfile
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/user/name/200/sendProfile/restart
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/phoneGroup/id/18/sendProfile
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/phoneGroup/id/18/sendProfile/restart
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/phoneGroup/name/phgr1/sendProfile
curl --insecure -X PUT https://superadmin:11111111@localhost/sipxconfig/api/phones/phoneGroup/name/phgr1/sendProfile/restart
EnhancementAPI
SIPX-719Add search phone based on user / line to REST APIAdd ability to retrieve phones for an user / line through REST APIEnhancementAPI
SIPX-720REST API - Add ability to search users based on emailImprove users rest api to retrieve users based on email field.EnhancementAPI
UC-4627REST API improvement to change group firmware for phone groupsA user would like to change group firmware for phone groups via REST calls.EnhancementAPI
UC-4686REST API improvement: change multiple settings with one rest callA customer would like to edit phone groups more efficiently via API.

A single setting for a phone group can be performed. For example:
https://localhost/sipxconfig/api/phoneGroups/34/model/polycom450/settings/lcl/time/24HourClock
with data 1 or 0 and it will update that one setting. What would be helpful would be to update multiple settings in a single API call.

Now to change multiple phone group settings in a call:
curl -X PUT --insecure -H "Content-Type: application/json" -d '{"settings":[{"path":"lcl/time/24HourClock","value":"0"},{"path":"lcl/datetime/date.format","value":"Md,D XX"}]}' https://superadmin:11111111@localhost/sipxconfig/api/phoneGroups/phgr1/model/polycom330/settings
EnhancementAPI
UC-4721When restoring a backup made on a previous version on another cluster, the servers are not correctly added

A bug has been verified in 17.10 and 18.04 when trying to restore a backup from a previous version ofcode.

To reproduce -

  1. Create a backuponcluster 1 running version 16.12 of UniteMe.
  2. Restore the backup on cluster 2 running version 18.04 of UniteMe. When doing this, the system -> servers primary is correctly kept, but the servers 2 and 3 are replaced with the ones in 16.12 and hence loose connection becoming uninitialized. Also, system -> databasesdoesnot load anymore. Checking rs.status() shows correctly the databases on the nodes before the restore. Also, modifying the domain and IP of the secondary nodes to the correct ones does not work.

WORKAROUND:
1. Prepare the 18.04 system for restoring from 16.12. Delete the secondary and arbiter databases.
2. Restore the config file. After the restore process is complete, the secondary servers will be replaced by the one on the original system.
3. Delete the secondary servers.
4. Re-add the secondary servers. sipxecs-setup to reset the servers and re-assign them with the new IDs provided.
5. After the servers are configured, go to system -> databases and add the secondaries to the system.

Known IssueBackup
SIPX-717CallQueue Default for RecordingChange CallQueue Default for Recording to Disabled.

At present default is Enabled and recording directory is blank which can cause a problem.
EnhancementCallQueue
UC-4684Call queue MoH replaces default MoH

Uploading CallQueue MoH seems to override the system moh setting. Navigating to features -> moh and applying default.wav works to correct, however after the default.wav plays the call queue MoH is next in line.


For existing queues customer must: 
1. Move all queue audio files from /var/sipxdata/mediaserver/data/moh to /var/sipxdata/mediaserver/data/prompts (use mv comand) 
or 
2. Upload the Queue Audio files again in UI 
and then manually delete old files from /var/sipxdata/mediaserver/data/moh

FixCallQueue
UC-4687Download entire CDR reports (currently limited to 5MB)An administrator is trying to pull yearly CDR reports via the historic page. No matter what time range is set once it is downloaded file is always MAX 5MB and only shows few days.EnhancementCDR
UC-4695When generating very large CSV history or reports, web interface crashesThis has occured on a server containing ~2.000.000 calls in history. When trying to generate either a history CSV either a report with all the calls, the client browser receives error 502 - Proxy Error. However, the process is still running in the background taking up resources, generating a file that will not be able to be downloaded. The web GUI cannot be contacted anymore and sipxconfig will need a restart.

This would definitely be a problem on single node, or if the proxy, registrar and other services are running on the same node.

The system the error occured in is on AWS and contains dual Xeon processors and 8 GB of ram. The system generates a csv with up to 1.000.000 entries in around 20 seconds.

25.000 entries - 4.3 MB
200.000 entries - 35 MB
300.000 entries - 52 MB
500.000 entries - 88 MB
1.000.000 entries - 174 MB
Over 1.100.000 entries the system cannot be contacted anymore.

The same problem goes for reports as well, system stops responding when you try to generate a report from a time period that has more than 2.000.000 entries.
FixCDR
UC-4549SipDiagram Builder: Create archive to view diagrams offlineAn administrator would like to be able to view SipDiagram Builder related information offline. Create an email-ready zip archive with all required files to be able view diagram offline.

The files are generated in the location /var/www/html/siplogviewer/tempdata/CALL_ID.
EnhancementDiagnostics
UC-4622Administrator DashboardAn Administrator would like to be able to see performance information about the servers operating in their Uniteme system.

sipXproxy Performance Information - SIP Messages per second, Queue Depth, Auth Cache Queue entries
sipregistrar Performance Information - # Registrations, # Registration requests per second
MongoDB Performance Information - reads / sec, writes / sec, etc.
System Performance - Packets / sec, CPU use, memory use, etc.

Display last X messages from important log files.

It's not required to build this dashboard in the current admin gui.
EnhancementDiagnostics
UC-4644Add additional data to Proxy Statistics ManagerExtend the new StatisticsManager service with additional metrics we need. Make all information available in the existing .json file.

Count the different SIP Methods, SIP Error Responses and to who, Top SIP 10 talkers overall, and additional information that would be useful for performance monitoring (tbd).

{"timestamp": "2018-02-16T15:50:54.088061Z", "REGISTERi": 110}
{"timestamp": "2018-02-16T15:50:54.088076Z", "REGISTERiF": 110}
{"timestamp": "2018-02-16T15:50:54.088091Z", "REGISTERiS": 0}
{"timestamp": "2018-02-16T15:50:54.088101Z", "REGISTERo": 375}
{"timestamp": "2018-02-16T15:50:54.088111Z", "REGISTERoF": 110}
{"timestamp": "2018-02-16T15:50:54.088120Z", "REGISTERoS": 0}
{"timestamp": "2018-02-16T15:50:54.088129Z", "proxy_active_transaction_count": 5255}
{"timestamp": "2018-02-16T15:50:54.088138Z", "proxy_avg_dispatch_speed": 0}
{"timestamp": "2018-02-16T15:50:54.088158Z", "proxy_avg_entity_db_read": 0}
{"timestamp": "2018-02-16T15:50:54.088167Z", "proxy_avg_regdb_read": 0}
{"timestamp": "2018-02-16T15:50:54.088177Z", "proxy_msq_queue_size": 0}
{"timestamp": "2018-02-16T15:50:54.088186Z", "proxy_ua_queue_size": 0}
{"timestamp": "2018-02-16T15:50:54.088195Z", "queue_size": 93}
{"timestamp": "2018-02-16T15:50:54.088204Z", "reqi": 220}
{"timestamp": "2018-02-16T15:50:54.088213Z", "reqo": 1117}
{"timestamp": "2018-02-16T15:50:54.088222Z", "rspi": 247}
{"timestamp": "2018-02-16T15:50:54.088231Z", "rspo": 383}
{"timestamp": "2018-02-16T15:51:00.039017Z", "BYEi": 130}
{"timestamp": "2018-02-16T15:51:00.039045Z", "BYEiF": 109}
{"timestamp": "2018-02-16T15:51:00.039064Z", "BYEiS": 0}
{"timestamp": "2018-02-16T15:51:00.039075Z", "BYEo": 706}
{"timestamp": "2018-02-16T15:51:00.039084Z", "BYEoF": 109}
{"timestamp": "2018-02-16T15:51:00.039097Z", "BYEoS": 0}

Please note that the following metrics are presently constantly 0 : proxy_avg_dispatch_speed, proxy_avg_entity_db_read, proxy_avg_regdb_read, proxy_msq_queue_size, proxy_ua_queue_size.

New Proxy Statistics Manager can be disabled for higher performance. There is a small performance penalty for writing these statistics to the .json file.

EnhancementDiagnostics
SIPX-695LDAP improvementCurrently "Username strip" under LDAP management applies to all LDAP imports. An administrator would like for this to be associated with each LDAP server.EnhancementLDAP
UC-4645LDAP Username import manipulationAn administrator would like more flexibility to manipulate usernames during LDAP import.

1. Add the ability to strip characters from the imported username via regex
2. Add the ability to append a suffix to the imported username
3. Add the ability to prepend a prefix to the imported username
EnhancementLDAP
SIPX-707Upgrade MongoDB to 3.4 (or 3.6 if available)Why:
- WiredTiger storage, no allocation of disk
- no more collection global lock but document lock (should greatly improve performance, get benchmarks from older versions vs 18.04)
- better usage of CPU cores
- up to 50 nodes in replica set

ToDo:
- provide latest Mongodb RPMs on sipXcom install
- rework sipxecs-setup procedure
- adapt database management from sipXconfig
- adapt backup/restore procedures
- adapt local regions feature
- test basic functionality
- full regression testing & bug fixing
EnhancementMongo
SIPX-723Add possibility to specify urls in product copyright valueAn administrator would like to customizing skin (as documented in: http://wiki.sipxcom.org/display/sipXcom/Customizing+Colors%2C+Layout+and+Logo)

The administrator would like to allow the inserting of links in product.copyright value.
EnhancementPortal
SIPX-721Add option to disable Proxy DNS lookupsThis is a performance tweak that can make a big difference with the performance of very large systems.

- Add option in UI needed to disable registrar and MWI lookups from proxy
- In case option checked then generate forwardingrules.xml with IP:port instead DNS record. For Registrar that should be easy as proxy and registrar are automatically enabled on same node. Figure out IP address to point to for MWI server.
- By default, option is disabled
EnhancementProxy
UC-4640Make EntityDB cache expiration configurable (sip code part)The Proxy Service already has a caching mechanism already implemented. With this Jira we will implement a flexible way to configure the cache timeout used for packets received by the proxy without digest (meaning with no authorization yet). Today, the cache is hardcoded to 30 seconds but it can be set to several minutes depending on the configuration required.

Also to avoid a possible synchronization issue between an admin deleting a user and having the UA not yet removed from the cache (even it is not happening often), we may also consider a simple API call so that when the Administrator performs a delete action, we delete the associated entry in the cache (or maybe a special "SIP delete" message to proxy for example).

Additionally, run tests at various traffic levels starting at 30 seconds to 3 minutes in 30-second intervals, 5 minutes and 10 minutes to pattern behavior and find optimum setting.
EnhancementProxy
UC-4641Make EntityDB cache expiration configurable (webui part)Add tosipxproxy config file (/etc/sipxpbx/sipXproxy-config): name: SIPX_PROXY_ENTITY_CACHE_EXPIRE, default is 30.
Add to sipxregistrar config file (/etc/sipxpbx/registrar-config) : name: SIP_REGISTRAR_ENTITY_CACHE_EXPIRE, default is 30.

Actually this should be single setting "Entity Cache Expiration" on webui, but value should be extracted into 2 files.
EnhancementProxy
UC-4676Fix misleading log message queue is over half fullDuring investigations it turned out that this log is wrong and "max - 100" was a hardcodded and unused value. Determine and log proper message with correct message queue depth.

2018-01-24T13:23:27.343137Z":122375:KERNEL:WARNING:s1.phx2.ezuce.ro:SipClientTcp-21:7fb4a8089700:sipxproxy:"OsMsgQShared::doSendCore message queue 'SipClientTcp-20' is over half full - count = 6818 max = 100"
FixProxy
UC-4677Misleading mongodb slow read alarmsWhen the primary server was shutdown (so no mongo for processes to connect to) proxies on running nodes were given ALARM_MONGODB_SLOW_READ and some exact time, e.g. 814 ms. So this is not something reported by mongo (as a reminder mongodb already logs slow queries in mongodb.log) and there is the possibility that such alarms to be wrongly raised (as a consequence of slowly processing for example).

- Double check code and correct scenarios where these slow read reports could be wrongly reported
FixProxy
SIPX-722Registrar fails to start when 400k subscription in mongo databaseOn startup all components using SipSubscribeServer try to retrieve and load in memory all entries within database.

In case of SIPX-722 there were too many records to fit in returned cursor so that one failed with

"2018-01-31T11:00:03.256938Z":22:SIP:ERR:int.phx2.ezuce.ro:SipSubscribeServer-29:7f5b7de25700:sipxregistry:"SipSubscribeServer::initialize failed - retrying after 500 milliseconds"
"2018-01-31T11:00:03.860614Z":23:SIP:EMERG:int.phx2.ezuce.ro:SipSubscribeServer-29:7f5b7de25700:sipxregistry:"ALARM_MONGODB_SLOW_READ Last Mongo read took a long time: document: node.subscription delay: 103 milliseconds"

This caused no registrars to be able to start. This is an edge case that exposed the bug, however, using such approach of loading all subscribes from database when process starts (and assuming number of records can fit in mongo cursor) could in normal operation result in:
- slow operating of mongo database (restarting a process or trying to recover by restarting processes at the same time can nail down entire cluster)
- slow start of SIP processes using this approach (checking the code it seems like not only registrar is using same approach, but also MWI, proxy, RLS and SAA)

The query that used to return all registrations from MongoDB (see below) was removed
FixRegistrar
UC-4637Re-route SIP Subscription TrafficAt the moment all SUBSCRIBE messages which are not directed to MWI are explicitly sent to Registrar by default.
Registrar the forwards subscriptions to SSS using 302. This creates unnecessary traffic load on Registrar and underlying the mongo database.

We will change forwardingrules.xml in a way to forward subscriptions to SSS by default without making another hop with Registrar.

Forwarding rules part:
<!-- All other SUBSCRIBE requests go to the SIP registry service -->
<routeTo ruriParams="sipx-noroute=Voicemail">&lt;rr.HOSTNAME:0;transport=tcp;x-sipx-routetoreg&gt;</routeTo>

should be changed to
<!-- All other SUBSCRIBE requests go to the SSS service -->
<routeTo>&lt;HOSTNAME:5140;transport=tcp&gt;</routeTo>

where HOSTNAME is IP of SSS and 5140 is SSS default port
EnhancementRegistrar
SIPX-726Create proper compound indexes for subscriptions and registrar databasesDuring latest load / profiling it was noticed that some operations were reported slowly.

- for updating registrations bindings
- for updating subscriptions bindings

Comparing queries with collection indexes it can be seen that following fileds that are queried were not indexed:
- for registrar: contact, shardid
- for subscription: fromUri, callId, eventTypeKey

Used compound indexes, see https://docs.mongodb.com/manual/core/index-compound/
EnhancementRegistrar Mongo
SIPX-727Update registration bindings using upsert instead delete / insertUpdating registrations was made up from two operations: delete registration then update registration.
Subscription database is already using upsert call instead remove / update, change database for updating registrations:
https://github.com/sipXcom/sipxecs/blob/release-18.04/sipXcommserverLib/src/sipdb/SubscribeDB.cpp#L174
EnhancementRegistrar Mongo
SIPX-728Call ensureIndex only once for registrations and subscribtions databasesEach time a registration is updated / inserted, a call to ensure database indexes is performed. This could be resource consuming and affecting system performance.

More, starting with version 3.0 this call is deprecated and it uses createIndexes call behind the scene, see: https://docs.mongodb.com/manual/reference/method/db.collection.ensureIndex/#db.collection.ensureIndex

Remove ensureIndex from CRUD operations and keep it only when service starts up.
EnhancementRegistrar Mongo
UC-4415Openuc-saa service requires a restart after server rebootOpenUC-saa service does not work properly (presence not working) on fresh server start or after reboot, until service restart is issued.

Added back a restart of saa service after 5 minutes of server run time.
FixSAA
UC-4706SIP Status memory consumption increases infinitelyA memory leak was discovered in sip status - when stressing the system with many subscribes, SIP status keeps increasing in memory consumption. On the test system the consumption of SIP status increased to over 40% of the total memory (8 GB Ram) and system killed the mongod process to free space.FixSIPStatus
UC-4699Openuc-saa and sipxsss tail oplog using $or queryWhile profiling mongo database some slow queries were noticed.

Deactivating openuc-saa and sipxsss improves load testing results.

These slow queries are triggered by openuc-saa / sipxsss tailing oplog for getting updates when an user is changed (speed dial with presence added / deleted, shared user activated / deactivated). Problem was that openuc-saa / sipxsss mechanism scans the entire oplog (that can have millions of documents).

Change saa and sss to subscribe to a specific namespace (that should be the imdb database) to improve performance.
EnhancementSSS SAA
SIPX-725Mark selinux as enforcing.Currently /etc/selinux/config is disabled.

Some customers may require it to be enforcing for security reasons.
EnhancementSystem
UC-4660Sipxsupervisor enables nfs services alwaysIt seems that by default /usr/share/sipxecs/cfinputs/plugin.d/sipxtcpdumplog.cf enables nfs services by default, even if the tcpdump services are disabled in sipxconfig webui. This leaves the nfs rpc related services running and bound to ports.

As a workaround the administrator could disable rpcbind and rpcgssd services, moved sipxtcpdumplog.cf out of the plugin.d directory, and rebooted.
FixSystem
UC-4665Remove Reachme from UnitemeWith Reachme moving to a standalone product at the end of May we will remove Reachme from Uniteme installation.

This will require that we address removing Uniteme and components during upgrades. We'll also address any remove backup tasks, etc.
EnhancementSystem
  • No labels