Yes, allowing your children to live with another relative during your custodial time could affect your primary custody. Just because you have primary custody does not mean that you can decide where the children will live during that time – it merely means that they are authorized to live with you.
Courts prefer that whenever possible, a child be with a parent. If you know that you will be temporarily out of state, you should tell your former spouse about the move, and ask if there is an issue if the children stay with your mother during that time. If your former spouse does not mind, ask him or her to sign an agreement to that effect. If your former spouse has an objection to that arrangement, you two should attempt to work together to figure out a solution for the time you have to be out of state.
If you do decide to move out of state with your children, and you two cannot agree on where your child will live during that time period, you cannot take your child with you unless the other parent agrees. Under state law, both parents must consent for the child to move more than 50 miles away for any period longer than 60 days.
If you absolutely have to leave, and your spouse will not agree on your child care arrangements, you have a couple of options. You can allow your children to stay with your former spouse for that time period. You could also ask the court for permission to take your children with you. In addition, you could also ask the court to allow your mother or another relative to care for the children.